[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:27 dist-test-slave-kb1g krb5kdc[4981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:27 dist-test-slave-kb1g krb5kdc[4981](info): set up 2 sockets
Sep 25 22:49:27 dist-test-slave-kb1g krb5kdc[4981](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:49:27.979041 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:28.123980 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41413
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:28.232745 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:28.233042 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:28.233124 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:28.237076 4994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:28.237169 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:28.237206 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:28.237265 4994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:28.237305 4994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:28.242528 4994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:28.243649 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:28.244745 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:28.253273 4999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.253436 4994 server_base.cc:1034] running on GCE node
W20260925 22:49:28.253273 5000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:28.253347 5002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.254056 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:28.255190 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:28.256378 4994 hybrid_clock.cc:648] HybridClock initialized: now 1790376568256339 us; error 58 us; skew 500 ppm
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:28.259100 4994 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:28.260038 4994 webserver.cc:460] Webserver started at http://127.4.219.190:33571/ using document root <none> and password file <none>
I20260925 22:49:28.260674 4994 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:28.260769 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:28.261005 4994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:28.263019 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "4b977136175a4afb997ed7b9ffd4c8ce"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "c9ddad4f6b982fba7c414b00be5c9e34"
server_key_iv: "169d934e5d50ed33f4fed14daa7a3b19"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.263619 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "4b977136175a4afb997ed7b9ffd4c8ce"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "c9ddad4f6b982fba7c414b00be5c9e34"
server_key_iv: "169d934e5d50ed33f4fed14daa7a3b19"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.267697 4994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:49:28.270182 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.271354 4994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:28.271497 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "4b977136175a4afb997ed7b9ffd4c8ce"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "c9ddad4f6b982fba7c414b00be5c9e34"
server_key_iv: "169d934e5d50ed33f4fed14daa7a3b19"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.271610 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:28.298480 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:28.301729 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:28.301951 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:28.309523 4994 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41413
I20260925 22:49:28.309540 5061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41413 every 8 connection(s)
I20260925 22:49:28.310606 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:28.313402 5062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:28.319218 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce: Bootstrap starting.
I20260925 22:49:28.320493 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 4994
I20260925 22:49:28.320739 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:28.321889 5062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:28.321965 4974 external_mini_cluster.cc:1428] Setting key e3f7876541b20590566b612a9476b41e
I20260925 22:49:28.322686 5062 log.cc:824] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:28.325227 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce: No bootstrap required, opened a new log
I20260925 22:49:28.329770 5062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } }
I20260925 22:49:28.330013 5062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:28.330051 5062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b977136175a4afb997ed7b9ffd4c8ce, State: Initialized, Role: FOLLOWER
I20260925 22:49:28.330525 5062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } }
I20260925 22:49:28.330686 5062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:28.330753 5062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:28.330868 5062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:28.332151 5062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } }
I20260925 22:49:28.332607 5062 leader_election.cc:302] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b977136175a4afb997ed7b9ffd4c8ce; no voters:
I20260925 22:49:28.332962 5062 leader_election.cc:288] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:28.333035 5067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:28.333227 5067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [term 1 LEADER]: Becoming Leader. State: Replica: 4b977136175a4afb997ed7b9ffd4c8ce, State: Running, Role: LEADER
I20260925 22:49:28.333534 5067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } }
I20260925 22:49:28.334316 5062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:28.335268 5068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } } }
I20260925 22:49:28.335443 5068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:28.335726 5069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b977136175a4afb997ed7b9ffd4c8ce. Latest consensus state: current_term: 1 leader_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b977136175a4afb997ed7b9ffd4c8ce" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41413 } } }
I20260925 22:49:28.335834 5069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:28.336954 5076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:28.340576 5076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:28.346481 5076 catalog_manager.cc:1347] Generated new cluster ID: 5cd4f8dc95dd4784b3cda9a97ecb542c
I20260925 22:49:28.346621 5076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:28.348080 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.325383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.4.219.190:41413)
0925 22:49:28.325872 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:28.325889 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:28.327192 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 22:49:28.329957 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:28.329969 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:28.330407 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:28.330763 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:28.334025 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.335469 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.337292 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.338010 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.338180 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:28.342157 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:28.342163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:28.342176 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:28.342204 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:28.343990 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.344582 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.344589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.344594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.344759 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.345212 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.345216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.345217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.345765 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:28.345985 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:28.346685 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 22:49:28.347027 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260925 22:49:28.360827 5076 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:28.361806 5076 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:28.369325 5076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b977136175a4afb997ed7b9ffd4c8ce: Generated new TSK 0
I20260925 22:49:28.370151 5076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:28.400292 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:28.519874 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:28.520155 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:28.520201 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:28.524137 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:28.524202 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:28.524262 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:28.529587 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:28.530640 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:28.531625 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:28.541311 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.541510 5088 server_base.cc:1034] running on GCE node
W20260925 22:49:28.541325 5096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:28.541317 5094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.542006 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:28.542644 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:28.543826 5088 hybrid_clock.cc:648] HybridClock initialized: now 1790376568543793 us; error 46 us; skew 500 ppm
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:28.546525 5088 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:28.547457 5088 webserver.cc:460] Webserver started at http://127.4.219.129:42283/ using document root <none> and password file <none>
I20260925 22:49:28.548019 5088 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:28.548113 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:28.548342 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:28.550411 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "169035f10c4f4252a0f82d6bf733be0d"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "a5e6e5677864f663d77ecc2e374399e6"
server_key_iv: "2c7367f40c968cbea97c888a1ed2d97d"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.550980 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "169035f10c4f4252a0f82d6bf733be0d"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "a5e6e5677864f663d77ecc2e374399e6"
server_key_iv: "2c7367f40c968cbea97c888a1ed2d97d"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.554896 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:28.557304 5103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.558511 5088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:49:28.558655 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "169035f10c4f4252a0f82d6bf733be0d"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "a5e6e5677864f663d77ecc2e374399e6"
server_key_iv: "2c7367f40c968cbea97c888a1ed2d97d"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.558768 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:28.574785 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:28.577929 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:28.578085 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:28.578632 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:28.579428 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:28.579474 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.579514 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:28.579530 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.590427 5088 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:42455
I20260925 22:49:28.590528 5216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:42455 every 8 connection(s)
I20260925 22:49:28.591457 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:28.596827 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5088
I20260925 22:49:28.596921 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:28.597178 4974 external_mini_cluster.cc:1428] Setting key 8fcccf4d524edc49fd54e6041d69b3cc
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:28.606683 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.593530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:58735 (local address 127.4.219.190:41413)
0925 22:49:28.593695 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:28.593699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:28.595017 (+ 1318us) server_negotiation.cc:407] Connection header received
0925 22:49:28.595895 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:28.595897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:28.595950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:28.596027 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:28.597500 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.598190 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.599373 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.599605 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.599732 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:28.602632 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:28.602639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:28.602645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:28.602681 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:28.604464 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.605021 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.605026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.605029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.605121 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.605442 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.605445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.605447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.605711 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:28.605819 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:28.606318 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 22:49:28.606451 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:28.607487 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.593892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41413 (local address 127.4.219.129:58735)
0925 22:49:28.594823 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:28.594869 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:28.595700 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:28.596220 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:28.596227 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:28.596702 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:28.597326 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:28.597343 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.598322 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.598325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:28.599241 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:28.599249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.599769 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.599773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:28.599956 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:28.600681 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:28.600698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:28.602438 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:28.604635 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:28.604640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:28.604654 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:28.604915 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:28.605213 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:28.605215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:28.605216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:28.605348 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:28.605814 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:28.605819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:28.606061 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 22:49:28.606232 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:49:28.606499 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":544,"threads_started":1}
I20260925 22:49:28.609018 5217 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41413
I20260925 22:49:28.609337 5217 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:28.609927 5217 heartbeater.cc:511] Master 127.4.219.190:41413 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:28.611624 5026 ts_manager.cc:194] Registered new tserver with Master: 169035f10c4f4252a0f82d6bf733be0d (127.4.219.129:42455)
I20260925 22:49:28.612876 5026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:58735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:28.625917 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:28.743607 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:28.743882 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:28.743932 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:28.748800 5222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:28.748867 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:28.748934 5222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:28.754343 5222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:28.755421 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:28.756423 5222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:28.766119 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.766248 5222 server_base.cc:1034] running on GCE node
W20260925 22:49:28.766113 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:28.766119 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:28.766881 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:28.767472 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:28.768608 5222 hybrid_clock.cc:648] HybridClock initialized: now 1790376568768573 us; error 45 us; skew 500 ppm
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:28.771330 5222 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:28.772197 5222 webserver.cc:460] Webserver started at http://127.4.219.130:35295/ using document root <none> and password file <none>
I20260925 22:49:28.772794 5222 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:28.772855 5222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:28.773020 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:28.775099 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "0cf7b6fea89a4d299c04b2ed72604bd0"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "002124aaddbf7cd0bfbe81434a650f0d"
server_key_iv: "b93f557314c2c28e743988ac200327ee"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.775748 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "0cf7b6fea89a4d299c04b2ed72604bd0"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "002124aaddbf7cd0bfbe81434a650f0d"
server_key_iv: "b93f557314c2c28e743988ac200327ee"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.779500 5222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:28.782037 5237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.783317 5222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:28.783468 5222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "0cf7b6fea89a4d299c04b2ed72604bd0"
format_stamp: "Formatted at 2026-09-25 22:49:28 on dist-test-slave-kb1g"
server_key: "002124aaddbf7cd0bfbe81434a650f0d"
server_key_iv: "b93f557314c2c28e743988ac200327ee"
server_key_version: "encryptionkey@0"
I20260925 22:49:28.783583 5222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:28.808884 5222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:28.811867 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:28.812083 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:28.812722 5222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:28.813586 5222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:28.813632 5222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.813707 5222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:28.813740 5222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:28.824167 5222 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40073
I20260925 22:49:28.824409 5350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40073 every 8 connection(s)
I20260925 22:49:28.825268 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:28.832857 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5222
I20260925 22:49:28.832962 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:28.833230 4974 external_mini_cluster.cc:1428] Setting key 2a0b0e80f79556fa9594ab69604f2527
Sep 25 22:49:28 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376568, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:28.840893 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.827258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38801 (local address 127.4.219.190:41413)
0925 22:49:28.827417 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:28.827422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:28.828385 (+ 963us) server_negotiation.cc:407] Connection header received
0925 22:49:28.829845 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:28.829848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:28.829898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:28.829980 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:28.831240 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.831775 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.833172 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.833453 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.833567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:28.836567 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:28.836572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:28.836576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:28.836603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:28.838112 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.838834 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.838843 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.838846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.838949 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:28.839478 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:28.839482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:28.839483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:28.839701 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:28.839799 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:28.840640 (+ 841us) server_negotiation.cc:299] Negotiation successful
0925 22:49:28.840757 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:28.842176 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:28.827743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41413 (local address 127.4.219.130:38801)
0925 22:49:28.828218 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:28.828263 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:28.829622 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:28.830199 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:28.830207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:28.830603 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:28.831101 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:28.831115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.831907 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.831916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:28.833043 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:28.833060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:28.833608 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:28.833612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:28.833792 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:28.834375 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:28.834394 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:28.836301 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:28.838245 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:28.838254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:28.838274 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:28.838672 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:28.839064 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:28.839069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:28.839071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:28.839287 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:28.839835 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:28.839845 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:28.840251 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 22:49:28.840506 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 22:49:28.840975 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:49:28.843580 5351 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41413
I20260925 22:49:28.843990 5351 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:28.844573 5351 heartbeater.cc:511] Master 127.4.219.190:41413 requested a full tablet report, sending...
I20260925 22:49:28.845718 5026 ts_manager.cc:194] Registered new tserver with Master: 0cf7b6fea89a4d299c04b2ed72604bd0 (127.4.219.130:40073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:49:28.846326 5026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:38801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:28.863551 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:28.978811 5356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:28.979055 5356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:28.979099 5356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:28.983069 5356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:28.983139 5356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:28.983199 5356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:28.988761 5356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41413
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:28.989885 5356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:28.990890 5356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:29.000901 5364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:29.000905 5362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:29.000906 5361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:29.001222 5356 server_base.cc:1034] running on GCE node
I20260925 22:49:29.001688 5356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:29.002347 5356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:29.003532 5356 hybrid_clock.cc:648] HybridClock initialized: now 1790376569003505 us; error 36 us; skew 500 ppm
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:29.006639 5356 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:29.007602 5356 webserver.cc:460] Webserver started at http://127.4.219.131:39337/ using document root <none> and password file <none>
I20260925 22:49:29.008164 5356 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:29.008225 5356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:29.008399 5356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:29.010711 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "7b3b55aababa42f9a9b874034a780492"
format_stamp: "Formatted at 2026-09-25 22:49:29 on dist-test-slave-kb1g"
server_key: "554b21c83fd2457b3b1ec86b804be225"
server_key_iv: "3d59169fbef0f0022a71546566032dda"
server_key_version: "encryptionkey@0"
I20260925 22:49:29.011269 5356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "7b3b55aababa42f9a9b874034a780492"
format_stamp: "Formatted at 2026-09-25 22:49:29 on dist-test-slave-kb1g"
server_key: "554b21c83fd2457b3b1ec86b804be225"
server_key_iv: "3d59169fbef0f0022a71546566032dda"
server_key_version: "encryptionkey@0"
I20260925 22:49:29.015323 5356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 22:49:29.018215 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:29.019572 5356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:49:29.019718 5356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "7b3b55aababa42f9a9b874034a780492"
format_stamp: "Formatted at 2026-09-25 22:49:29 on dist-test-slave-kb1g"
server_key: "554b21c83fd2457b3b1ec86b804be225"
server_key_iv: "3d59169fbef0f0022a71546566032dda"
server_key_version: "encryptionkey@0"
I20260925 22:49:29.019838 5356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:29.045467 5356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:29.048578 5356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:29.048794 5356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:29.049417 5356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:29.050318 5356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:29.050390 5356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:29.050495 5356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:29.050546 5356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:29.060200 5356 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:44575
I20260925 22:49:29.060222 5484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:44575 every 8 connection(s)
I20260925 22:49:29.061194 5356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:29.070143 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5356
I20260925 22:49:29.070291 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:29.070593 4974 external_mini_cluster.cc:1428] Setting key 7f610be215f86f511134e241aa61c80f
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:29.075416 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.063113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:41685 (local address 127.4.219.190:41413)
0925 22:49:29.063269 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:29.063272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:29.064132 (+ 860us) server_negotiation.cc:407] Connection header received
0925 22:49:29.065492 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:29.065495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:29.065542 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:29.065616 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:29.066732 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.067248 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.068339 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.068604 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.068686 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:29.071738 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:29.071741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:29.071744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:29.071770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:29.073552 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.074012 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.074014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.074015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.074087 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.074370 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.074372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.074373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.074536 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:29.074609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:29.075106 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 22:49:29.075208 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:49:29.076344 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.063543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41413 (local address 127.4.219.131:41685)
0925 22:49:29.063975 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:29.064016 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:29.065298 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:29.065791 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:29.065798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:29.066121 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:29.066576 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:29.066589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.067350 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.067353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:29.068238 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:29.068246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.068731 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.068734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:29.068871 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:29.069485 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:29.069501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:29.071583 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:29.073669 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:29.073673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:29.073685 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:29.073928 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:29.074169 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:29.074171 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:29.074172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:29.074295 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:29.074618 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:29.074623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:29.074906 (+ 283us) client_negotiation.cc:769] Sending connection context
0925 22:49:29.075059 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:49:29.075366 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260925 22:49:29.077432 5485 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41413
I20260925 22:49:29.077672 5485 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:29.078130 5485 heartbeater.cc:511] Master 127.4.219.190:41413 requested a full tablet report, sending...
I20260925 22:49:29.079093 5026 ts_manager.cc:194] Registered new tserver with Master: 7b3b55aababa42f9a9b874034a780492 (127.4.219.131:44575)
I20260925 22:49:29.079643 5026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:41685
I20260925 22:49:29.085069 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:29.113162 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.103529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.4.219.129:42455)
0925 22:49:29.103876 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:29.103891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:29.103918 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:49:29.104046 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:29.104053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:29.104262 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:29.104396 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:29.105160 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.105963 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.106808 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.107217 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.107310 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:29.109717 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:29.109726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:29.109740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:29.109762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:29.111487 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.111871 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.111876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.111877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.111951 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.112175 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.112178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.112179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.112406 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:29.112621 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:29.112802 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:49:29.112962 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
W20260925 22:49:29.114220 5131 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55292
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[4981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:29.141450 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:29.133054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55294 (local address 127.4.219.129:42455)
0925 22:49:29.133197 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:29.133201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:29.133382 (+ 181us) server_negotiation.cc:407] Connection header received
0925 22:49:29.133562 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:29.133565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:29.133607 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:29.133706 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:29.134509 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.135032 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.135964 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:29.136231 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:29.136327 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:29.138634 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:29.138637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:29.138639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:29.138662 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:29.140054 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.140406 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.140408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.140409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.140478 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:29.140774 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:29.140776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:29.140777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:29.140935 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:29.141033 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:29.141232 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:49:29.141327 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:29.142596 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5088
I20260925 22:49:29.150270 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5222
I20260925 22:49:29.156368 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5356
I20260925 22:49:29.161809 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 4994
2026-09-25T22:49:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2235 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[5499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[5499](info): set up 2 sockets
Sep 25 22:49:29 dist-test-slave-kb1g krb5kdc[5499](info): commencing operation
krb5kdc: starting...
W20260925 22:49:30.200322 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:30.323529 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44219
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:30.439549 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:30.439888 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:30.439971 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:30.444188 5512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:30.444296 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:30.444335 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:30.444372 5512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:30.444412 5512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:30.450085 5512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:30.451272 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:30.452376 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:30.461228 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.461228 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.461221 5517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:30.461277 5512 server_base.cc:1034] running on GCE node
I20260925 22:49:30.462018 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:30.463181 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:30.464396 5512 hybrid_clock.cc:648] HybridClock initialized: now 1790376570464385 us; error 34 us; skew 500 ppm
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:30.467247 5512 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:30.468174 5512 webserver.cc:460] Webserver started at http://127.4.219.190:36893/ using document root <none> and password file <none>
I20260925 22:49:30.468793 5512 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:30.468855 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:30.469046 5512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:30.471140 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "ae8ac4480a4149bd8650d3525e7536cc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "12300c782025b673e242e29a48041269"
server_key_iv: "d0d9224c376ce5a92b7f3ad43f278759"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.471666 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "ae8ac4480a4149bd8650d3525e7536cc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "12300c782025b673e242e29a48041269"
server_key_iv: "d0d9224c376ce5a92b7f3ad43f278759"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.475409 5512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:30.477905 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:30.479200 5512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:30.479346 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "ae8ac4480a4149bd8650d3525e7536cc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "12300c782025b673e242e29a48041269"
server_key_iv: "d0d9224c376ce5a92b7f3ad43f278759"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.479454 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:30.503834 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:30.507475 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:30.507822 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:30.515777 5512 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44219
I20260925 22:49:30.515812 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44219 every 8 connection(s)
I20260925 22:49:30.516882 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:30.519639 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:30.520102 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5512
I20260925 22:49:30.520222 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:30.520506 4974 external_mini_cluster.cc:1428] Setting key 381a26520a0f9c59c868c8b0622e3843
I20260925 22:49:30.525779 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc: Bootstrap starting.
I20260925 22:49:30.528450 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:30.529402 5580 log.cc:824] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:30.531999 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc: No bootstrap required, opened a new log
I20260925 22:49:30.535830 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.522098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.4.219.190:44219)
0925 22:49:30.522541 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:30.522556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:30.522592 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:49:30.523280 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:30.523298 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:30.523658 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:30.523963 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:30.524823 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.525949 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.527095 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.527566 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.527688 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:30.530703 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:30.530709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:30.530726 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:30.530755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:30.532745 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:30.533176 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:30.533182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:30.533188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:30.533323 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:30.533582 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:30.533585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:30.533587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:30.534113 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:30.534338 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:30.534545 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:49:30.534867 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260925 22:49:30.536651 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } }
I20260925 22:49:30.536875 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:30.536934 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8ac4480a4149bd8650d3525e7536cc, State: Initialized, Role: FOLLOWER
I20260925 22:49:30.537370 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } }
I20260925 22:49:30.537531 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:30.537616 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:30.537736 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:30.538797 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } }
I20260925 22:49:30.539142 5580 leader_election.cc:302] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae8ac4480a4149bd8650d3525e7536cc; no voters:
I20260925 22:49:30.539451 5580 leader_election.cc:288] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:30.539557 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:30.539815 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [term 1 LEADER]: Becoming Leader. State: Replica: ae8ac4480a4149bd8650d3525e7536cc, State: Running, Role: LEADER
I20260925 22:49:30.540189 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } }
I20260925 22:49:30.540742 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:30.541884 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae8ac4480a4149bd8650d3525e7536cc. Latest consensus state: current_term: 1 leader_uuid: "ae8ac4480a4149bd8650d3525e7536cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } } }
I20260925 22:49:30.541944 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae8ac4480a4149bd8650d3525e7536cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae8ac4480a4149bd8650d3525e7536cc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44219 } } }
I20260925 22:49:30.542029 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:30.542035 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:30.542466 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:30.545634 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:30.551014 5594 catalog_manager.cc:1347] Generated new cluster ID: 923807eec9dc48ae9817a9c8ee774cfb
I20260925 22:49:30.551091 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:30.572675 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:30.573973 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:30.582018 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae8ac4480a4149bd8650d3525e7536cc: Generated new TSK 0
I20260925 22:49:30.582877 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:30.613579 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:30.724300 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:30.724619 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:30.724697 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:30.728699 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:30.728789 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:30.728880 5606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:30.734004 5606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:30.735072 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:30.736097 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:30.745568 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.745618 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.745635 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:30.745851 5606 server_base.cc:1034] running on GCE node
I20260925 22:49:30.746243 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:30.746838 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:30.747988 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790376570747970 us; error 31 us; skew 500 ppm
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:30.750708 5606 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:30.751528 5606 webserver.cc:460] Webserver started at http://127.4.219.129:33177/ using document root <none> and password file <none>
I20260925 22:49:30.752076 5606 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:30.752136 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:30.752345 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:30.754407 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "baf822288e8741b1ad49791162329dbc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "a2fd9b17b54958ea7e6fb07221625f30"
server_key_iv: "d9fe4ef7065cbdfb534e32971ff41a0a"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.754972 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "baf822288e8741b1ad49791162329dbc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "a2fd9b17b54958ea7e6fb07221625f30"
server_key_iv: "d9fe4ef7065cbdfb534e32971ff41a0a"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.758549 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:30.760908 5621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:30.762089 5606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:30.762204 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "baf822288e8741b1ad49791162329dbc"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "a2fd9b17b54958ea7e6fb07221625f30"
server_key_iv: "d9fe4ef7065cbdfb534e32971ff41a0a"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.762301 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:30.789292 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:30.792248 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:30.792449 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:30.793057 5606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:30.793880 5606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:30.793948 5606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:30.794036 5606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:30.794071 5606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:30.802891 5606 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:34045
I20260925 22:49:30.802913 5734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:34045 every 8 connection(s)
I20260925 22:49:30.803790 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:30.810381 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5606
I20260925 22:49:30.810474 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:30.810726 4974 external_mini_cluster.cc:1428] Setting key 88d7b13d9f6372c054459a580b48751a
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:30.818029 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.805703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:51795 (local address 127.4.219.190:44219)
0925 22:49:30.805841 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:30.805844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:30.806698 (+ 854us) server_negotiation.cc:407] Connection header received
0925 22:49:30.808077 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:30.808081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:30.808150 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:30.808242 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:30.809502 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.810036 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.811431 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.811664 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.811777 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:30.814310 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:30.814318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:30.814321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:30.814345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:30.815772 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:30.816406 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:30.816408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:30.816409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:30.816481 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:30.816893 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:30.816896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:30.816897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:30.817069 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:30.817156 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:30.817811 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 22:49:30.817907 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:30.819293 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:30.806092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44219 (local address 127.4.219.129:51795)
0925 22:49:30.806551 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:30.806594 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:30.807837 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:30.808414 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:30.808421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:30.808849 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:30.809342 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:30.809355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.810162 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.810166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:30.811310 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:30.811320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:30.811810 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:30.811814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:30.812035 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:30.812717 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:30.812734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:30.814132 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:30.815890 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:30.815897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:30.815914 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:30.816307 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:30.816597 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:30.816603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:30.816608 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:30.816797 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:30.817181 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:30.817190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:30.817540 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 22:49:30.817774 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:49:30.818149 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:30.820734 5735 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44219
I20260925 22:49:30.820982 5735 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:30.821559 5735 heartbeater.cc:511] Master 127.4.219.190:44219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:30.823117 5544 ts_manager.cc:194] Registered new tserver with Master: baf822288e8741b1ad49791162329dbc (127.4.219.129:34045)
I20260925 22:49:30.824265 5544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:51795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:30.837594 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:30.947294 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:30.947566 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:30.947610 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:30.951603 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:30.951661 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:30.951719 5740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:30.957051 5740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:30.958078 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:30.959043 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:30.968667 5740 server_base.cc:1034] running on GCE node
W20260925 22:49:30.968674 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.968627 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:30.968837 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:30.969205 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:30.969851 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:30.971078 5740 hybrid_clock.cc:648] HybridClock initialized: now 1790376570971035 us; error 52 us; skew 500 ppm
Sep 25 22:49:30 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:30.973824 5740 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:30.974718 5740 webserver.cc:460] Webserver started at http://127.4.219.130:40839/ using document root <none> and password file <none>
I20260925 22:49:30.975273 5740 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:30.975360 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:30.975592 5740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:30.977701 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "53177c8bc65647e0aff31ac7af595de7"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "75a1a6d794a6542b490dd524e4da2991"
server_key_iv: "55b73c03d78d9058a5da6d9e570a0108"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.978242 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "53177c8bc65647e0aff31ac7af595de7"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "75a1a6d794a6542b490dd524e4da2991"
server_key_iv: "55b73c03d78d9058a5da6d9e570a0108"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.981866 5740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:30.984395 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:30.985749 5740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:30.985874 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "53177c8bc65647e0aff31ac7af595de7"
format_stamp: "Formatted at 2026-09-25 22:49:30 on dist-test-slave-kb1g"
server_key: "75a1a6d794a6542b490dd524e4da2991"
server_key_iv: "55b73c03d78d9058a5da6d9e570a0108"
server_key_version: "encryptionkey@0"
I20260925 22:49:30.985984 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:31.004285 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:31.007277 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:31.007479 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:31.008071 5740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:31.008913 5740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:31.008982 5740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:31.009053 5740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:31.009105 5740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:31.017905 5740 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:35115
I20260925 22:49:31.017930 5868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:35115 every 8 connection(s)
I20260925 22:49:31.018774 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:31.023780 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5740
I20260925 22:49:31.023916 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:31.024189 4974 external_mini_cluster.cc:1428] Setting key 5f8b8cfdbe8c7e016327ff0ecef003bb
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:31.033771 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.020481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:47257 (local address 127.4.219.190:44219)
0925 22:49:31.020637 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.020640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.021538 (+ 898us) server_negotiation.cc:407] Connection header received
0925 22:49:31.022872 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.022876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.022934 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.023041 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.024284 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.024953 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.026214 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.026434 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.026542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.029683 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.029686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.029689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.029714 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.031535 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.032047 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.032051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.032053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.032150 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.032465 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.032468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.032470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.032771 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.032883 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.033378 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.033521 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:49:31.034497 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.020871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44219 (local address 127.4.219.130:47257)
0925 22:49:31.021388 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.021428 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.022657 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.023222 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.023229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.023579 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:31.024143 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.024159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.025071 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.025076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.026071 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.026080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.026578 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.026582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.026758 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.027396 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:31.027421 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:31.029340 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:31.031671 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:31.031676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:31.031688 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:31.031939 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:31.032243 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:31.032245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:31.032246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:31.032374 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:31.032876 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:31.032883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:31.033125 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.033285 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.033538 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":172,"threads_started":1}
I20260925 22:49:31.035754 5869 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44219
I20260925 22:49:31.036013 5869 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:49:31.036510 5869 heartbeater.cc:511] Master 127.4.219.190:44219 requested a full tablet report, sending...
I20260925 22:49:31.037629 5544 ts_manager.cc:194] Registered new tserver with Master: 53177c8bc65647e0aff31ac7af595de7 (127.4.219.130:35115)
I20260925 22:49:31.038326 5544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:47257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:31.053544 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:31.164923 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:31.165207 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:31.165278 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:31.169448 5874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:31.169529 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:31.169617 5874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:31.174803 5874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:31.175836 5874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:31.176923 5874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:31.186592 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:31.186681 5874 server_base.cc:1034] running on GCE node
W20260925 22:49:31.186612 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:31.186592 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:31.187306 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:31.187955 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:31.189278 5874 hybrid_clock.cc:648] HybridClock initialized: now 1790376571189265 us; error 28 us; skew 500 ppm
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:31.192060 5874 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:31.192982 5874 webserver.cc:460] Webserver started at http://127.4.219.131:40119/ using document root <none> and password file <none>
I20260925 22:49:31.193533 5874 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:31.193640 5874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:31.193869 5874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:31.195888 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347"
format_stamp: "Formatted at 2026-09-25 22:49:31 on dist-test-slave-kb1g"
server_key: "d3bae6c207d435ece7b4019ba9fe86c8"
server_key_iv: "949a1d79f8a4e2e993ab3e5570bc64f4"
server_key_version: "encryptionkey@0"
I20260925 22:49:31.196506 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347"
format_stamp: "Formatted at 2026-09-25 22:49:31 on dist-test-slave-kb1g"
server_key: "d3bae6c207d435ece7b4019ba9fe86c8"
server_key_iv: "949a1d79f8a4e2e993ab3e5570bc64f4"
server_key_version: "encryptionkey@0"
I20260925 22:49:31.200311 5874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:31.202786 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:31.204041 5874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:31.204176 5874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347"
format_stamp: "Formatted at 2026-09-25 22:49:31 on dist-test-slave-kb1g"
server_key: "d3bae6c207d435ece7b4019ba9fe86c8"
server_key_iv: "949a1d79f8a4e2e993ab3e5570bc64f4"
server_key_version: "encryptionkey@0"
I20260925 22:49:31.204290 5874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:31.233891 5874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:31.237059 5874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:31.237283 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:31.237905 5874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:31.238747 5874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:31.238818 5874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:31.238891 5874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:31.238934 5874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:31.248129 5874 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:41579
I20260925 22:49:31.248157 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:41579 every 8 connection(s)
I20260925 22:49:31.249142 5874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:31.249912 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 5874
I20260925 22:49:31.250038 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:31.250365 4974 external_mini_cluster.cc:1428] Setting key f990cce82dfe1fc6cd9e2bb183d4ace2
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:31.263171 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.251215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:57427 (local address 127.4.219.190:44219)
0925 22:49:31.251364 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.251368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.252293 (+ 925us) server_negotiation.cc:407] Connection header received
0925 22:49:31.253763 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.253766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.253819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.253895 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.255009 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.255541 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.256716 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.256948 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.257054 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.259367 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.259371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.259374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.259400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.261126 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.261638 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.261642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.261644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.261726 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.262054 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.262059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.262061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.262253 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.262357 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.262850 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.262995 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:49:31.264178 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.251681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44219 (local address 127.4.219.131:57427)
0925 22:49:31.252155 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.252197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.253551 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.254061 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.254069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.254408 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:31.254875 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.254887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.255663 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.255666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.256598 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.256607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.257088 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.257093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.257280 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.257822 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:31.257837 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:31.259212 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:31.261249 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:31.261254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:31.261278 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:31.261519 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:31.261814 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:31.261818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:31.261820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:31.261957 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:31.262355 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:31.262362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:31.262616 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.262802 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.263132 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":151,"threads_started":1}
I20260925 22:49:31.265381 6003 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44219
I20260925 22:49:31.265632 6003 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:31.266153 6003 heartbeater.cc:511] Master 127.4.219.190:44219 requested a full tablet report, sending...
I20260925 22:49:31.267150 5544 ts_manager.cc:194] Registered new tserver with Master: 7a523e0a1c2a4a58ab2030f7c8cd6347 (127.4.219.131:41579)
I20260925 22:49:31.267719 5544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:57427
I20260925 22:49:31.276511 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:31.287431 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.280082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.4.219.190:44219)
0925 22:49:31.280232 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.280240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.280255 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:31.280328 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.280330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.280370 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.280487 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.281214 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.281822 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.282628 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.282862 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.282974 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.284566 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.284571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.284574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.284605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.285921 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.286304 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.286306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.286307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.286379 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.286635 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.286637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.286638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.286865 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.286977 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.287200 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.287300 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:49:31.292433 5544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:31.295442 5544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:31.310962 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41579 (local address 127.0.0.1:33612)
0925 22:49:31.305363 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.305417 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.305555 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.306300 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.306302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.306316 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:31.306409 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.306418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307993 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.307996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.309248 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.309258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.310337 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.310463 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.310506 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.310637 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.310769 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"spinlock_wait_cycles":563584,"thread_start_us":83,"threads_started":1}
I20260925 22:49:31.311125 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.4.219.129:34045)
0925 22:49:31.305237 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.305250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.305914 (+ 664us) server_negotiation.cc:407] Connection header received
0925 22:49:31.305996 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.306001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.306167 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.306300 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:31.306679 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307516 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.309653 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.310520 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310651 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.310783 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.310880 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.310974 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":61,"threads_started":1}
I20260925 22:49:31.311121 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34045 (local address 127.0.0.1:33942)
0925 22:49:31.305805 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.305818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.305908 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.306453 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.306457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.306476 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:31.306561 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.306565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307645 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.307651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.309527 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.309533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.310652 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.310746 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.310768 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.310849 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.310933 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":702,"thread_start_us":90,"threads_started":1}
I20260925 22:49:31.311964 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.4.219.131:41579)
0925 22:49:31.305135 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.305146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.305518 (+ 372us) server_negotiation.cc:407] Connection header received
0925 22:49:31.305777 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.305782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.305994 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.306159 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:31.306511 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307666 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.309382 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.310205 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.310479 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.311725 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.311828 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":174,"threads_started":1}
I20260925 22:49:31.312588 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.4.219.130:35115)
0925 22:49:31.305413 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.305425 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.305573 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:49:31.305726 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.305732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.305971 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.306132 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:31.306439 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307642 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310057 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.310806 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.310904 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.311031 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.312342 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.312426 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":136,"threads_started":1}
I20260925 22:49:31.312795 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.304490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:35115 (local address 127.0.0.1:39662)
0925 22:49:31.305363 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.305422 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.305558 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.306144 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.306149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.306171 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:31.306315 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.306322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.307786 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.307790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.309952 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.309957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.311113 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.311118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.311208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.311228 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.311415 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.311503 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"spinlock_wait_cycles":1408,"thread_start_us":110,"threads_started":1}
I20260925 22:49:31.314666 5669 tablet_service.cc:1467] Processing CreateTablet for tablet af3cc60b2dba4869995e6c7b516922d5 (DEFAULT_TABLE table=test-table [id=3837a3cbc3224b38b672f2c9faaf9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:31.314666 5937 tablet_service.cc:1467] Processing CreateTablet for tablet af3cc60b2dba4869995e6c7b516922d5 (DEFAULT_TABLE table=test-table [id=3837a3cbc3224b38b672f2c9faaf9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:31.315058 5803 tablet_service.cc:1467] Processing CreateTablet for tablet af3cc60b2dba4869995e6c7b516922d5 (DEFAULT_TABLE table=test-table [id=3837a3cbc3224b38b672f2c9faaf9065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:31.315864 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3cc60b2dba4869995e6c7b516922d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:31.315864 5669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3cc60b2dba4869995e6c7b516922d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:31.316035 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3cc60b2dba4869995e6c7b516922d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:31.322633 6020 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: Bootstrap starting.
I20260925 22:49:31.323949 6021 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: Bootstrap starting.
I20260925 22:49:31.324383 6022 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: Bootstrap starting.
I20260925 22:49:31.326151 6020 tablet_bootstrap.cc:654] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:31.326346 6022 tablet_bootstrap.cc:654] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:31.326666 6021 tablet_bootstrap.cc:654] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:31.326982 6020 log.cc:824] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:31.327531 6022 log.cc:824] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:31.327522 6021 log.cc:824] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:31.329107 6020 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: No bootstrap required, opened a new log
I20260925 22:49:31.329298 6022 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: No bootstrap required, opened a new log
I20260925 22:49:31.329308 6020 ts_tablet_manager.cc:1397] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:49:31.329391 6021 tablet_bootstrap.cc:492] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: No bootstrap required, opened a new log
I20260925 22:49:31.329486 6022 ts_tablet_manager.cc:1397] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:31.329595 6021 ts_tablet_manager.cc:1397] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:49:31.332968 6022 raft_consensus.cc:348] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.333243 6022 raft_consensus.cc:374] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:31.333325 6022 raft_consensus.cc:729] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a523e0a1c2a4a58ab2030f7c8cd6347, State: Initialized, Role: FOLLOWER
I20260925 22:49:31.333384 6021 raft_consensus.cc:348] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.333686 6021 raft_consensus.cc:374] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:31.333755 6021 raft_consensus.cc:729] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53177c8bc65647e0aff31ac7af595de7, State: Initialized, Role: FOLLOWER
I20260925 22:49:31.333847 6022 consensus_queue.cc:260] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.333971 6020 raft_consensus.cc:348] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.334170 6021 consensus_queue.cc:260] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.334270 6020 raft_consensus.cc:374] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:31.334383 6020 raft_consensus.cc:729] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf822288e8741b1ad49791162329dbc, State: Initialized, Role: FOLLOWER
I20260925 22:49:31.334506 6003 heartbeater.cc:503] Master 127.4.219.190:44219 was elected leader, sending a full tablet report...
I20260925 22:49:31.334882 6022 ts_tablet_manager.cc:1428] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:31.334879 6020 consensus_queue.cc:260] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.335146 6021 ts_tablet_manager.cc:1428] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:31.335451 5869 heartbeater.cc:503] Master 127.4.219.190:44219 was elected leader, sending a full tablet report...
I20260925 22:49:31.335500 5735 heartbeater.cc:503] Master 127.4.219.190:44219 was elected leader, sending a full tablet report...
I20260925 22:49:31.335750 6020 ts_tablet_manager.cc:1428] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260925 22:49:31.371026 6027 raft_consensus.cc:482] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:31.371244 6027 raft_consensus.cc:504] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.372474 6027 leader_election.cc:288] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf822288e8741b1ad49791162329dbc (127.4.219.129:34045), 7a523e0a1c2a4a58ab2030f7c8cd6347 (127.4.219.131:41579)
I20260925 22:49:31.377559 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.372789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34045 (local address 127.4.219.130:49649)
0925 22:49:31.373250 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.373260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.373330 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.373551 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.373555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.373566 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:31.373732 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.373737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.374773 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.374776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.376130 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.376141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.377121 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.377125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.377297 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.377319 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.377382 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.377435 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":182,"thread_start_us":125,"threads_started":1}
I20260925 22:49:31.377662 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.373033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49649 (local address 127.4.219.129:34045)
0925 22:49:31.373220 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.373225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.373337 (+ 112us) server_negotiation.cc:407] Connection header received
0925 22:49:31.373397 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.373400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.373457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.373533 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:31.373834 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.374679 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.376271 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.376974 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.377107 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.377163 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.377466 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.377537 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 22:49:31.377663 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.372782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41579 (local address 127.4.219.130:41563)
0925 22:49:31.372981 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:31.372996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:31.373085 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:31.373423 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:31.373426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:31.373439 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:31.373595 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.373602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.374682 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.374684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.376130 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:31.376141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.377125 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.377128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:31.377290 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.377316 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:49:31.377413 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:49:31.377496 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":90}
I20260925 22:49:31.378168 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.372874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:41563 (local address 127.4.219.131:41579)
0925 22:49:31.373021 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.373027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.373044 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:49:31.373225 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.373229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.373313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.373405 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:31.373700 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.374568 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.376271 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.376990 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.377108 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.377178 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.377970 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.378035 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:49:31.378437 5689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3cc60b2dba4869995e6c7b516922d5" candidate_uuid: "53177c8bc65647e0aff31ac7af595de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf822288e8741b1ad49791162329dbc" is_pre_election: true
I20260925 22:49:31.378747 5957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3cc60b2dba4869995e6c7b516922d5" candidate_uuid: "53177c8bc65647e0aff31ac7af595de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" is_pre_election: true
I20260925 22:49:31.378799 5689 raft_consensus.cc:2454] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53177c8bc65647e0aff31ac7af595de7 in term 0.
I20260925 22:49:31.379029 5957 raft_consensus.cc:2454] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53177c8bc65647e0aff31ac7af595de7 in term 0.
I20260925 22:49:31.379367 5759 leader_election.cc:302] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53177c8bc65647e0aff31ac7af595de7, baf822288e8741b1ad49791162329dbc; no voters:
I20260925 22:49:31.379691 6027 raft_consensus.cc:2781] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:31.379779 6027 raft_consensus.cc:482] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:31.379838 6027 raft_consensus.cc:3037] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:31.381067 6027 raft_consensus.cc:504] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.381477 6027 leader_election.cc:288] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [CANDIDATE]: Term 1 election: Requested vote from peers baf822288e8741b1ad49791162329dbc (127.4.219.129:34045), 7a523e0a1c2a4a58ab2030f7c8cd6347 (127.4.219.131:41579)
I20260925 22:49:31.381791 5689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3cc60b2dba4869995e6c7b516922d5" candidate_uuid: "53177c8bc65647e0aff31ac7af595de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf822288e8741b1ad49791162329dbc"
I20260925 22:49:31.381835 5957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3cc60b2dba4869995e6c7b516922d5" candidate_uuid: "53177c8bc65647e0aff31ac7af595de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347"
I20260925 22:49:31.381937 5689 raft_consensus.cc:3037] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:31.381943 5957 raft_consensus.cc:3037] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:31.383451 5957 raft_consensus.cc:2454] T af3cc60b2dba4869995e6c7b516922d5 P 7a523e0a1c2a4a58ab2030f7c8cd6347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53177c8bc65647e0aff31ac7af595de7 in term 1.
I20260925 22:49:31.383474 5689 raft_consensus.cc:2454] T af3cc60b2dba4869995e6c7b516922d5 P baf822288e8741b1ad49791162329dbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53177c8bc65647e0aff31ac7af595de7 in term 1.
I20260925 22:49:31.383770 5756 leader_election.cc:302] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53177c8bc65647e0aff31ac7af595de7, 7a523e0a1c2a4a58ab2030f7c8cd6347; no voters:
I20260925 22:49:31.383962 6027 raft_consensus.cc:2781] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:31.384177 6027 raft_consensus.cc:686] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [term 1 LEADER]: Becoming Leader. State: Replica: 53177c8bc65647e0aff31ac7af595de7, State: Running, Role: LEADER
I20260925 22:49:31.384505 6027 consensus_queue.cc:237] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } }
I20260925 22:49:31.387738 5544 catalog_manager.cc:5495] T af3cc60b2dba4869995e6c7b516922d5 P 53177c8bc65647e0aff31ac7af595de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53177c8bc65647e0aff31ac7af595de7 (127.4.219.130). New cstate: current_term: 1 leader_uuid: "53177c8bc65647e0aff31ac7af595de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baf822288e8741b1ad49791162329dbc" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a523e0a1c2a4a58ab2030f7c8cd6347" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53177c8bc65647e0aff31ac7af595de7" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 35115 } health_report { overall_health: HEALTHY } } }
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:31.422443 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.409066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.4.219.129:34045)
0925 22:49:31.409609 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.409617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.409634 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:49:31.409677 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.409680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.409730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.409821 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.410537 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.411098 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.411983 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.412221 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.412334 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.417923 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.417968 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.417990 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.418024 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.420422 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.420995 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.421004 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.421006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.421100 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.421401 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.421405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.421406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.421676 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.421916 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.422114 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.422240 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:31.455046 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.445705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.4.219.129:34045)
0925 22:49:31.445874 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.445880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.446127 (+ 247us) server_negotiation.cc:407] Connection header received
0925 22:49:31.446267 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.446270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.446330 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.446433 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.447337 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.448099 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.449181 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.449412 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.449494 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.451892 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.451895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.451897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.451922 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.453490 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.453878 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.453880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.453882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.453951 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.454220 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.454222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.454223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.454373 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.454467 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.454785 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.454884 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[5499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:31.481418 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:31.472460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.4.219.129:34045)
0925 22:49:31.472753 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:31.472758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:31.472966 (+ 208us) server_negotiation.cc:407] Connection header received
0925 22:49:31.473027 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:31.473030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:31.473080 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:31.473160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:31.473829 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.474337 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.475156 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:31.475388 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:31.475533 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:31.478184 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:31.478188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:31.478190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:31.478220 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:31.479633 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.480066 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.480070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.480071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.480185 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:31.480433 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:31.480435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:31.480436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:31.480678 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:31.480778 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:31.481050 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 22:49:31.481211 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
W20260925 22:49:31.482911 5649 tablet_service.cc:2908] Rejecting scan request for tablet af3cc60b2dba4869995e6c7b516922d5: Uninitialized: clean time has not yet been initialized
I20260925 22:49:31.485523 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5606
I20260925 22:49:31.492422 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5740
I20260925 22:49:31.498311 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5874
I20260925 22:49:31.503648 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 5512
2026-09-25T22:49:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2341 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[6044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[6044](info): set up 2 sockets
Sep 25 22:49:31 dist-test-slave-kb1g krb5kdc[6044](info): commencing operation
krb5kdc: starting...
W20260925 22:49:32.548441 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:32 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:32.670928 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:38009
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:38009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:32.779615 6057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:32.779888 6057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:32.779927 6057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:32.784056 6057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:32.784117 6057 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:49:32.784134 6057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:32.784152 6057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:32.784168 6057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:32.784183 6057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:32.789516 6057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:38009
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:32.790521 6057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:32.791481 6057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:32.799902 6062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:32.799893 6065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:32.799893 6063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:32.800120 6057 server_base.cc:1034] running on GCE node
I20260925 22:49:32.800792 6057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:32.801872 6057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:32.803063 6057 hybrid_clock.cc:648] HybridClock initialized: now 1790376572803033 us; error 44 us; skew 500 ppm
Sep 25 22:49:32 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376572, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:32.812429 6057 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:32.813346 6057 webserver.cc:460] Webserver started at http://127.4.219.190:42277/ using document root <none> and password file <none>
I20260925 22:49:32.813858 6057 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:32.813915 6057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:32.814087 6057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:32.816071 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "90e574a4d5b544e483486695375b0537"
format_stamp: "Formatted at 2026-09-25 22:49:32 on dist-test-slave-kb1g"
server_key: "fb0d37f91ffcdca8ae2a8fcdc5e32396"
server_key_iv: "5dc31321eb8edb9ac48ae94bb547a1ce"
server_key_version: "encryptionkey@0"
I20260925 22:49:32.816736 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "90e574a4d5b544e483486695375b0537"
format_stamp: "Formatted at 2026-09-25 22:49:32 on dist-test-slave-kb1g"
server_key: "fb0d37f91ffcdca8ae2a8fcdc5e32396"
server_key_iv: "5dc31321eb8edb9ac48ae94bb547a1ce"
server_key_version: "encryptionkey@0"
I20260925 22:49:32.820422 6057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:49:32.822818 6072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:32.824069 6057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:32.824165 6057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "90e574a4d5b544e483486695375b0537"
format_stamp: "Formatted at 2026-09-25 22:49:32 on dist-test-slave-kb1g"
server_key: "fb0d37f91ffcdca8ae2a8fcdc5e32396"
server_key_iv: "5dc31321eb8edb9ac48ae94bb547a1ce"
server_key_version: "encryptionkey@0"
I20260925 22:49:32.824249 6057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:32.837484 6057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:32.840736 6057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:32.840955 6057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:32.849290 6057 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:38009
I20260925 22:49:32.849305 6134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:38009 every 8 connection(s)
I20260925 22:49:32.850229 6057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:32.853044 6135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:32.857021 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6057
I20260925 22:49:32.857115 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:32.857362 4974 external_mini_cluster.cc:1428] Setting key d1271dd335d6f6828400a5e7efc909bc
I20260925 22:49:32.858939 6135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Bootstrap starting.
I20260925 22:49:32.861323 6135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:32.862056 6135 log.cc:824] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:32.863911 6135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: No bootstrap required, opened a new log
Sep 25 22:49:32 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:32.867285 6135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } }
I20260925 22:49:32.867532 6135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:32.867626 6135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e574a4d5b544e483486695375b0537, State: Initialized, Role: FOLLOWER
I20260925 22:49:32.868131 6135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } }
I20260925 22:49:32.868260 6135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:32.868350 6135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:32.868475 6135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:32.869354 6135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } }
I20260925 22:49:32.869699 6135 leader_election.cc:302] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e574a4d5b544e483486695375b0537; no voters:
I20260925 22:49:32.870009 6135 leader_election.cc:288] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:32.870139 6140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:32.870388 6140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [term 1 LEADER]: Becoming Leader. State: Replica: 90e574a4d5b544e483486695375b0537, State: Running, Role: LEADER
I20260925 22:49:32.870726 6140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } }
I20260925 22:49:32.871305 6135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:32.871845 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:32.858811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.4.219.190:38009)
0925 22:49:32.859273 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:32.859288 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:32.859325 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:49:32.859996 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:32.860012 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:32.860441 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:32.860829 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:32.861614 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:32.862773 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:32.863634 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:32.864145 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:32.864308 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:32.866731 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:32.866739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:32.866757 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:32.866783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:32.868687 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:32.869120 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:32.869126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:32.869131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:32.869246 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:32.869529 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:32.869533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:32.869535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:32.869976 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:32.870192 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:32.870593 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 22:49:32.870897 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":144,"threads_started":1}
I20260925 22:49:32.872388 6142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e574a4d5b544e483486695375b0537. Latest consensus state: current_term: 1 leader_uuid: "90e574a4d5b544e483486695375b0537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } } }
I20260925 22:49:32.872499 6142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:32.872898 6149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:32.873731 6141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e574a4d5b544e483486695375b0537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e574a4d5b544e483486695375b0537" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 38009 } } }
I20260925 22:49:32.873826 6141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:32.876092 6149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:32.882014 6149 catalog_manager.cc:1347] Generated new cluster ID: 7492d02274f64ad0ae4c13eb80f2a62a
I20260925 22:49:32.882124 6149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:32.893271 6149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:32.894212 6149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:32.902360 6149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Generated new TSK 0
I20260925 22:49:32.903065 6149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:32.931805 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:38009
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:33.041863 6161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:33.042109 6161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:33.042150 6161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:33.046218 6161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:33.046280 6161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:33.046347 6161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:33.046429 6161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:33.051659 6161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:33.052772 6161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:33.053785 6161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:33.063077 6169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:33.063081 6166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:33.063077 6167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:33.063443 6161 server_base.cc:1034] running on GCE node
I20260925 22:49:33.063869 6161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:33.064452 6161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:33.065637 6161 hybrid_clock.cc:648] HybridClock initialized: now 1790376573065609 us; error 41 us; skew 500 ppm
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:33.068351 6161 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:33.069296 6161 webserver.cc:460] Webserver started at http://127.4.219.129:46489/ using document root <none> and password file <none>
I20260925 22:49:33.069842 6161 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:33.069932 6161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:33.070158 6161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:33.072214 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "7c069c3e421840057d71953438f606d5"
server_key_iv: "e4d26115fa1d50e751ab6bc8a259bde0"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.072782 6161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "7c069c3e421840057d71953438f606d5"
server_key_iv: "e4d26115fa1d50e751ab6bc8a259bde0"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.076203 6161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:49:33.078559 6176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.079703 6161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:33.079818 6161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "7c069c3e421840057d71953438f606d5"
server_key_iv: "e4d26115fa1d50e751ab6bc8a259bde0"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.079916 6161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:33.096428 6161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:33.099764 6161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:33.099937 6161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:33.101562 6161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:33.101646 6161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.101718 6161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:33.101778 6161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:33.112782 6161 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:34497
I20260925 22:49:33.112829 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:34497 every 8 connection(s)
I20260925 22:49:33.113934 6161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:33.117098 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.102367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:32819 (local address 127.4.219.190:38009)
0925 22:49:33.102521 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.102527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.103450 (+ 923us) server_negotiation.cc:407] Connection header received
0925 22:49:33.104590 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.104597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.104668 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.104785 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.106673 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.107604 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.109714 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.109942 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.110030 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.113217 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.113221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.113224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.113260 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.114989 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.115528 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.115531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.115532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.115612 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.115964 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.115966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.115967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.116150 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.116226 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.116869 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.116974 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:49:33.117950 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6161
I20260925 22:49:33.118060 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:33.118080 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.102819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.129:32819)
0925 22:49:33.103261 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.103311 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.104311 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.105169 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.105182 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.105705 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.106507 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.106534 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.107755 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.107762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.109593 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.109605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.110134 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.110140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.110372 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.111127 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.111149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.113035 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.115111 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.115119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.115138 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.115424 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.115718 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.115721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.115722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.115878 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.116273 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.116279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.116629 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.116815 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.117075 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":137,"threads_started":1}
I20260925 22:49:33.118356 4974 external_mini_cluster.cc:1428] Setting key 562cb61468326a2f575bbf1e12dc2cff
I20260925 22:49:33.119426 6292 heartbeater.cc:347] Connected to a master server at 127.4.219.190:38009
I20260925 22:49:33.119655 6292 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:33.120153 6292 heartbeater.cc:511] Master 127.4.219.190:38009 requested a full tablet report, sending...
I20260925 22:49:33.121801 6089 ts_manager.cc:194] Registered new tserver with Master: c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497)
I20260925 22:49:33.123136 6089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:32819
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:33.130228 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.121967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:39635 (local address 127.4.219.190:38009)
0925 22:49:33.122104 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.122109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.122388 (+ 279us) server_negotiation.cc:407] Connection header received
0925 22:49:33.122503 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.122506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.122552 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.122637 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.123230 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.123947 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.125105 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.125547 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.125682 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.127126 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.127129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.127131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.127152 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.128701 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.129195 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.129198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.129199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.129271 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.129511 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.129513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.129514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.129667 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.129738 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.129931 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.130068 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:33.130241 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.121888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.129:39635)
0925 22:49:33.122279 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.122297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.122409 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.122678 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.122683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.122928 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.123098 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.123108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.124077 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.124102 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.124973 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.124983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.125724 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.125731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.125823 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.126475 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.126491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.126991 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.128806 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.128809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.128811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.129096 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.129366 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.129368 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.129370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.129447 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.129744 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.129746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.129818 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.129895 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.130098 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:33.147049 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:38009
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:33.255065 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:33.255326 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:33.255370 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:33.259358 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:33.259416 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:33.259455 6301 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:33.259527 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:33.264873 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:33.265942 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:33.266999 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:33.276599 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:33.276611 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:33.276652 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:33.276945 6301 server_base.cc:1034] running on GCE node
I20260925 22:49:33.277426 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:33.278070 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:33.279259 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790376573279203 us; error 69 us; skew 500 ppm
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:33.282208 6301 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:33.283282 6301 webserver.cc:460] Webserver started at http://127.4.219.130:42839/ using document root <none> and password file <none>
I20260925 22:49:33.283970 6301 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:33.284068 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:33.284283 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:33.286412 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "f92a7b5baef442628cf4173c7af62baf"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "3daca158493cd92aed66d91adfb18fa2"
server_key_iv: "a9b5518407d2158477019e0a69ab3e30"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.286983 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "f92a7b5baef442628cf4173c7af62baf"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "3daca158493cd92aed66d91adfb18fa2"
server_key_iv: "a9b5518407d2158477019e0a69ab3e30"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.290789 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:33.293417 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.294664 6301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:33.294804 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "f92a7b5baef442628cf4173c7af62baf"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "3daca158493cd92aed66d91adfb18fa2"
server_key_iv: "a9b5518407d2158477019e0a69ab3e30"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.294922 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:33.309463 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:33.312510 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:33.312775 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:33.314231 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:33.314302 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.314378 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:33.314428 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:33.325235 6301 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:34643
I20260925 22:49:33.325289 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:34643 every 8 connection(s)
I20260925 22:49:33.326179 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:33.328090 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.315032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:46803 (local address 127.4.219.190:38009)
0925 22:49:33.315212 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.315219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.316024 (+ 805us) server_negotiation.cc:407] Connection header received
0925 22:49:33.317183 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.317189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.317259 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.317361 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.318632 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.319222 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.320620 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.320894 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.321003 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.324155 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.324159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.324162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.324191 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.325825 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.326347 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.326351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.326352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.326442 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.326988 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.326991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.326993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.327223 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.327361 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.327853 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.327973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:49:33.328998 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.315410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.130:46803)
0925 22:49:33.315856 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.315905 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.316940 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.317568 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.317595 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.317951 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.318468 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.318488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.319349 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.319359 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.320453 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.320469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.321059 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.321063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.321225 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.321846 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.321862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.323992 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.325955 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.325961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.325977 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.326236 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.326599 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.326604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.326606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.326854 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.327352 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.327359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.327611 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.327774 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.328024 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":142,"threads_started":1}
I20260925 22:49:33.330170 6432 heartbeater.cc:347] Connected to a master server at 127.4.219.190:38009
I20260925 22:49:33.330328 6432 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:33.330808 6432 heartbeater.cc:511] Master 127.4.219.190:38009 requested a full tablet report, sending...
I20260925 22:49:33.331843 6088 ts_manager.cc:194] Registered new tserver with Master: f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:33.332664 6088 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:46803
I20260925 22:49:33.333240 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6301
I20260925 22:49:33.333349 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:33.333642 4974 external_mini_cluster.cc:1428] Setting key 17868b726316f300c74cf330f59ba588
I20260925 22:49:33.339804 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.332285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:41727 (local address 127.4.219.190:38009)
0925 22:49:33.332400 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.332403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.332597 (+ 194us) server_negotiation.cc:407] Connection header received
0925 22:49:33.332778 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.332784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.332829 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.332927 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.333410 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.333910 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.335109 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.335378 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.335494 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.336557 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.336563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.336569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.336600 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.338039 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.338521 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.338524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.338525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.338595 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.338879 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.338881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.338882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.339131 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.339211 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.339594 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.339694 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:49:33.339896 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.332224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.130:41727)
0925 22:49:33.332504 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.332517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.332661 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.332950 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.332954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.333175 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.333300 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.333306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.334074 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.334077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.334972 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.334984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.335488 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.335490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.335568 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.336091 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.336102 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.336391 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.338145 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.338151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.338156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.338417 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.338682 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.338688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.338693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.338793 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.339233 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.339239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.339354 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.339476 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.339642 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:33.359560 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:38009
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:33.470464 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:33.470717 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:33.470759 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:33.475378 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:33.475435 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:33.475505 6441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:33.475587 6441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:33.480772 6441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:38009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:33.481850 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:33.482931 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:33.492686 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:33.492674 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:33.492733 6441 server_base.cc:1034] running on GCE node
W20260925 22:49:33.492674 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:33.493407 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:33.494076 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:33.495241 6441 hybrid_clock.cc:648] HybridClock initialized: now 1790376573495207 us; error 45 us; skew 500 ppm
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:33.498168 6441 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:33.499248 6441 webserver.cc:460] Webserver started at http://127.4.219.131:34695/ using document root <none> and password file <none>
I20260925 22:49:33.499817 6441 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:33.499877 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:33.500054 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:33.502269 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "e738df10ca1b49479db4d0c4aadbb911"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "c7d174d04f1a828be85779673e54599b"
server_key_iv: "3a4433e23e37a95c3f940f2975041eba"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.502827 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "e738df10ca1b49479db4d0c4aadbb911"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "c7d174d04f1a828be85779673e54599b"
server_key_iv: "3a4433e23e37a95c3f940f2975041eba"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.506695 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:33.509150 6456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.510385 6441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:33.510542 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "e738df10ca1b49479db4d0c4aadbb911"
format_stamp: "Formatted at 2026-09-25 22:49:33 on dist-test-slave-kb1g"
server_key: "c7d174d04f1a828be85779673e54599b"
server_key_iv: "3a4433e23e37a95c3f940f2975041eba"
server_key_version: "encryptionkey@0"
I20260925 22:49:33.510658 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:33.545579 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:33.548678 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:33.548890 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:33.550340 6441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:33.550419 6441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:33.550501 6441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:33.550549 6441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:33.561502 6441 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:37933
I20260925 22:49:33.561573 6571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:37933 every 8 connection(s)
I20260925 22:49:33.562650 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:33.564855 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.551398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45579 (local address 127.4.219.190:38009)
0925 22:49:33.551537 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.551542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.552727 (+ 1185us) server_negotiation.cc:407] Connection header received
0925 22:49:33.553869 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.553873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.553968 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.554060 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.555646 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.556571 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.557963 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.558201 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.558319 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.560986 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.560993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.560998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.561032 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.562742 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.563327 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.563330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.563332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.563437 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.563833 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.563837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.563838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.564050 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.564138 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.564637 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.564740 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:49:33.565701 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.551989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.131:45579)
0925 22:49:33.552501 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.552622 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.553640 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.554263 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.554272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.554717 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.555455 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.555477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.556711 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.556714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.557820 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.557829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.558372 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.558376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.558540 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.559152 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.559168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.560798 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.562845 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.562849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.562860 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.563209 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.563545 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.563548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.563550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.563710 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.564147 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.564152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.564360 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.564577 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.564832 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260925 22:49:33.565737 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6441
I20260925 22:49:33.565828 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:33.566107 4974 external_mini_cluster.cc:1428] Setting key edfb5efa6530a8a1c27d534d147e73b1
I20260925 22:49:33.567077 6572 heartbeater.cc:347] Connected to a master server at 127.4.219.190:38009
I20260925 22:49:33.567859 6572 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:33.568364 6572 heartbeater.cc:511] Master 127.4.219.190:38009 requested a full tablet report, sending...
I20260925 22:49:33.569398 6089 ts_manager.cc:194] Registered new tserver with Master: e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131:37933)
I20260925 22:49:33.570205 6089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:45579
I20260925 22:49:33.575816 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.569227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:47539 (local address 127.4.219.190:38009)
0925 22:49:33.569352 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.569356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.569538 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:49:33.569653 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.569655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.569691 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.569751 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.570217 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.570819 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.571731 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.572052 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.572153 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.573345 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.573348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.573349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.573368 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.574554 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.574921 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.574924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.574925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.574992 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.575212 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.575214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.575215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.575358 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.575429 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.575634 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.575716 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:33.575817 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.569174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.4.219.131:47539)
0925 22:49:33.569468 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.569483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.569575 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.569792 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.569795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.570012 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:33.570117 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.570123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.570940 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.570943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.571567 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.571575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.572165 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.572168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.572240 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.572803 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:33.572825 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:33.573237 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:33.574647 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.574649 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.574651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.574841 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.575067 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:33.575069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:33.575071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:33.575139 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:33.575433 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:33.575435 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:33.575503 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.575579 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.575675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260925 22:49:33.579602 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:49:33 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:33.616015 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.606034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.4.219.190:38009)
0925 22:49:33.606257 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.606264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.606288 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:49:33.606372 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.606377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.606465 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.606596 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:33.607806 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.608794 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.609660 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.610078 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.610198 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.612705 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:33.612708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:33.612710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:33.612737 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:33.614099 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.614600 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.614603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.614605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.614686 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:33.615011 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:33.615013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:33.615015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:33.615223 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:33.615331 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.615688 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.615877 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:33.619130 6089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:33.622323 6089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:33.636897 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.631877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.4.219.130:34643)
0925 22:49:33.632398 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.632409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.632428 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:33.632501 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.632505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.632693 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.632855 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:33.633061 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.634222 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.635527 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.636218 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.636342 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.636520 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.636657 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.636743 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":120,"threads_started":1}
I20260925 22:49:33.636888 6591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.631135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:34643 (local address 127.0.0.1:54310)
0925 22:49:33.632177 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.632190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.632268 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.632833 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.632836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.632847 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:33.632950 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.632954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.634360 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.634364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.635393 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.635403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.636341 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.636345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.636448 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.636486 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.636628 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.636744 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":21,"thread_start_us":89,"threads_started":1}
I20260925 22:49:33.638288 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.631564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:37933 (local address 127.0.0.1:60918)
0925 22:49:33.632621 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.632635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.632730 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.633625 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.633627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.633640 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:33.633705 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.633710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.634807 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.634810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.635770 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.635780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.637176 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.637180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.637991 (+ 811us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.638019 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.638093 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.638161 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1001,"thread_start_us":78,"threads_started":1}
I20260925 22:49:33.638779 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.631138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.4.219.129:34497)
0925 22:49:33.631414 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.631422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.632038 (+ 616us) server_negotiation.cc:407] Connection header received
0925 22:49:33.632354 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.632360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.632599 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.632757 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:33.633061 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.634233 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.636673 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.637456 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.637562 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.637684 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.638559 (+ 875us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.638665 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
I20260925 22:49:33.638805 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.632248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.4.219.131:37933)
0925 22:49:33.632611 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:33.632622 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:33.633067 (+ 445us) server_negotiation.cc:407] Connection header received
0925 22:49:33.633141 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:33.633144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:33.633344 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:33.633498 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:33.633814 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.634693 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.635896 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.636695 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.636783 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.636938 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:33.638576 (+ 1638us) server_negotiation.cc:299] Negotiation successful
0925 22:49:33.638673 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260925 22:49:33.638463 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:33.630983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34497 (local address 127.0.0.1:57130)
0925 22:49:33.631894 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:33.631944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:33.632101 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:33.632741 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:33.632749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:33.632775 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:33.632946 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.632953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.635508 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.635510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.636493 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:33.636503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:33.637598 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:33.637602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:33.638152 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:33.638185 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:49:33.638257 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:49:33.638330 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260925 22:49:33.640256 6366 tablet_service.cc:1467] Processing CreateTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (DEFAULT_TABLE table=test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:33.641492 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:33.641827 6506 tablet_service.cc:1467] Processing CreateTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (DEFAULT_TABLE table=test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:33.642849 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:33.642844 6225 tablet_service.cc:1467] Processing CreateTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (DEFAULT_TABLE table=test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:33.643698 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:33.650230 6595 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Bootstrap starting.
I20260925 22:49:33.650991 6596 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Bootstrap starting.
I20260925 22:49:33.651031 6597 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Bootstrap starting.
I20260925 22:49:33.652696 6595 tablet_bootstrap.cc:654] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:33.652731 6596 tablet_bootstrap.cc:654] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:33.652957 6597 tablet_bootstrap.cc:654] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:33.653507 6596 log.cc:824] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:33.653558 6595 log.cc:824] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:33.653630 6597 log.cc:824] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:33.655454 6597 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: No bootstrap required, opened a new log
I20260925 22:49:33.655519 6595 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: No bootstrap required, opened a new log
I20260925 22:49:33.655692 6596 tablet_bootstrap.cc:492] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: No bootstrap required, opened a new log
I20260925 22:49:33.655720 6595 ts_tablet_manager.cc:1397] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:49:33.655903 6596 ts_tablet_manager.cc:1397] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:49:33.656384 6597 ts_tablet_manager.cc:1397] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:49:33.659319 6597 raft_consensus.cc:348] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.659590 6597 raft_consensus.cc:374] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:33.659662 6597 raft_consensus.cc:729] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Initialized, Role: FOLLOWER
I20260925 22:49:33.659932 6596 raft_consensus.cc:348] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.660210 6596 raft_consensus.cc:374] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:33.660167 6597 consensus_queue.cc:260] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.660284 6596 raft_consensus.cc:729] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92a7b5baef442628cf4173c7af62baf, State: Initialized, Role: FOLLOWER
I20260925 22:49:33.660257 6595 raft_consensus.cc:348] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.660509 6595 raft_consensus.cc:374] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:33.660578 6595 raft_consensus.cc:729] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17ef7fb33d64daea19a9b368e0b3f18, State: Initialized, Role: FOLLOWER
I20260925 22:49:33.660746 6596 consensus_queue.cc:260] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.660989 6595 consensus_queue.cc:260] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:33.661329 6597 ts_tablet_manager.cc:1428] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:33.661445 6432 heartbeater.cc:503] Master 127.4.219.190:38009 was elected leader, sending a full tablet report...
I20260925 22:49:33.661689 6596 ts_tablet_manager.cc:1428] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:49:33.661702 6572 heartbeater.cc:503] Master 127.4.219.190:38009 was elected leader, sending a full tablet report...
I20260925 22:49:33.661919 6595 ts_tablet_manager.cc:1428] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 22:49:33.662233 6292 heartbeater.cc:503] Master 127.4.219.190:38009 was elected leader, sending a full tablet report...
W20260925 22:49:33.813714 6573 tablet.cc:2369] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:33.827407 6433 tablet.cc:2369] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:33.865468 6293 tablet.cc:2369] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:34.017871 6602 raft_consensus.cc:482] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:34.018110 6601 raft_consensus.cc:482] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:34.018261 6602 raft_consensus.cc:504] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.018349 6601 raft_consensus.cc:504] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.019958 6601 leader_election.cc:288] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.020268 6602 leader_election.cc:288] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131:37933)
I20260925 22:49:34.024762 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.020122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:45763 (local address 127.4.219.131:37933)
0925 22:49:34.020229 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.020235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.020367 (+ 132us) server_negotiation.cc:407] Connection header received
0925 22:49:34.020646 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.020651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.020698 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.020744 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:34.021146 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022015 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.023549 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.024125 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024216 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.024267 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.024593 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.024642 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":42}
I20260925 22:49:34.024988 6593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.020684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:59981 (local address 127.4.219.130:34643)
0925 22:49:34.020785 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.020790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.020807 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:49:34.020859 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.020865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.020929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.021002 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:34.021344 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022080 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.023549 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.024125 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024225 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.024267 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.024815 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.024886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:49:34.025125 6605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.019779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:37933 (local address 127.4.219.130:45763)
0925 22:49:34.020290 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.020303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.020408 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.020889 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.020892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.020911 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:34.021021 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.021032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022098 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.022101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.023409 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.023420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.024346 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.024438 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.024457 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.024886 (+ 429us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.024953 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260925 22:49:34.024762 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.019825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:34643 (local address 127.4.219.131:59981)
0925 22:49:34.019943 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.019960 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.020054 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.021102 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.021105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.021122 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:34.021231 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.021240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022167 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.022170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.023409 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.023427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.024318 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.024431 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.024450 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.024522 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.024627 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:49:34.025403 6604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.019779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34497 (local address 127.4.219.130:60573)
0925 22:49:34.020487 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.020502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.020600 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.021118 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.021123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.021141 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:34.021226 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.021231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022057 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.022063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.024083 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.024093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.025013 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.025017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.025112 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.025134 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.025181 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.025244 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":734,"thread_start_us":60,"threads_started":1}
I20260925 22:49:34.025766 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf" is_pre_election: true
I20260925 22:49:34.026021 6381 raft_consensus.cc:2454] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.026075 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.020417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:60573 (local address 127.4.219.129:34497)
0925 22:49:34.020872 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.020881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.020893 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:34.020951 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.020954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.021000 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.021318 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:34.021419 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.021939 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024222 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.024869 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.025768 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.025827 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.025865 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.025929 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":138,"threads_started":1}
I20260925 22:49:34.026176 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "f92a7b5baef442628cf4173c7af62baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e738df10ca1b49479db4d0c4aadbb911" is_pre_election: true
I20260925 22:49:34.026398 6526 raft_consensus.cc:2454] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f92a7b5baef442628cf4173c7af62baf in term 0.
I20260925 22:49:34.026503 6459 leader_election.cc:302] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e738df10ca1b49479db4d0c4aadbb911, f92a7b5baef442628cf4173c7af62baf; no voters:
I20260925 22:49:34.026750 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.020204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:42961 (local address 127.4.219.129:34497)
0925 22:49:34.020334 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.020338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.020875 (+ 537us) server_negotiation.cc:407] Connection header received
0925 22:49:34.021049 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.021051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.021095 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.021140 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:34.022226 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022697 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.024686 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.025216 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.025292 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.025335 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.026574 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.026650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:34.026770 6601 raft_consensus.cc:2781] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:34.026758 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "f92a7b5baef442628cf4173c7af62baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" is_pre_election: true
I20260925 22:49:34.026804 6317 leader_election.cc:302] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e738df10ca1b49479db4d0c4aadbb911, f92a7b5baef442628cf4173c7af62baf; no voters:
I20260925 22:49:34.026849 6601 raft_consensus.cc:482] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:34.026891 6601 raft_consensus.cc:3037] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.027060 6246 raft_consensus.cc:2454] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f92a7b5baef442628cf4173c7af62baf in term 0.
I20260925 22:49:34.027068 6602 raft_consensus.cc:2781] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:34.027143 6602 raft_consensus.cc:482] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:34.027192 6602 raft_consensus.cc:3037] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.027482 6606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.019825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34497 (local address 127.4.219.131:42961)
0925 22:49:34.020784 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.020798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.020937 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.021275 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.021280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.021300 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:34.022142 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.022148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.022812 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.022818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.024552 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.024562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.026035 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.026042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.026394 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.026428 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.027095 (+ 667us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.027289 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":318,"thread_start_us":73,"threads_started":1}
I20260925 22:49:34.028200 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" is_pre_election: true
I20260925 22:49:34.028332 6246 raft_consensus.cc:2454] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.028306 6601 raft_consensus.cc:504] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.028380 6602 raft_consensus.cc:504] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.028720 6602 leader_election.cc:288] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [CANDIDATE]: Term 1 election: Requested vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131:37933)
I20260925 22:49:34.028811 6601 leader_election.cc:288] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Requested vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.029042 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf"
I20260925 22:49:34.029098 6526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "f92a7b5baef442628cf4173c7af62baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e738df10ca1b49479db4d0c4aadbb911"
I20260925 22:49:34.029098 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
I20260925 22:49:34.029211 6381 raft_consensus.cc:2379] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e738df10ca1b49479db4d0c4aadbb911 in current term 1: Already voted for candidate f92a7b5baef442628cf4173c7af62baf in this term.
I20260925 22:49:34.029196 6246 raft_consensus.cc:3037] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.029246 6526 raft_consensus.cc:2379] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f92a7b5baef442628cf4173c7af62baf in current term 1: Already voted for candidate e738df10ca1b49479db4d0c4aadbb911 in this term.
I20260925 22:49:34.029296 6245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b56d5a1b0c4407087d9fa1233ca1e2c" candidate_uuid: "f92a7b5baef442628cf4173c7af62baf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
I20260925 22:49:34.029778 6318 leader_election.cc:302] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f92a7b5baef442628cf4173c7af62baf; no voters: c17ef7fb33d64daea19a9b368e0b3f18, e738df10ca1b49479db4d0c4aadbb911
I20260925 22:49:34.029945 6602 raft_consensus.cc:2726] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 22:49:34.030464 6246 raft_consensus.cc:2454] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 1.
I20260925 22:49:34.030740 6458 leader_election.cc:302] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c17ef7fb33d64daea19a9b368e0b3f18, e738df10ca1b49479db4d0c4aadbb911; no voters: f92a7b5baef442628cf4173c7af62baf
I20260925 22:49:34.030965 6601 raft_consensus.cc:2781] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:34.031191 6601 raft_consensus.cc:686] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 LEADER]: Becoming Leader. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Running, Role: LEADER
I20260925 22:49:34.031502 6601 consensus_queue.cc:237] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.034685 6089 catalog_manager.cc:5495] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 reported cstate change: term changed from 0 to 1, leader changed from <none> to e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:34.096359 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.092630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.4.219.131:37933)
0925 22:49:34.092747 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.092750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.092764 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:34.092932 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.092935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.093000 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.093093 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:34.093375 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.093867 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.094900 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.095121 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.095199 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.095498 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:34.095951 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:34.096058 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.096139 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.096186 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:49:34.102038 6246 raft_consensus.cc:1261] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.102072 6381 raft_consensus.cc:1261] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.102636 6608 consensus_queue.cc:1035] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.102805 6601 consensus_queue.cc:1035] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.109079 6611 mvcc.cc:205] Tried to move back new op lower bound from 7333382447515525120 to 7333382447235342336. Current Snapshot: MvccSnapshot[applied={T|T < 7333382447515525120}]
I20260925 22:49:34.110176 6613 mvcc.cc:205] Tried to move back new op lower bound from 7333382447515525120 to 7333382447235342336. Current Snapshot: MvccSnapshot[applied={T|T < 7333382447515525120}]
I20260925 22:49:34.113169 6612 mvcc.cc:205] Tried to move back new op lower bound from 7333382447515525120 to 7333382447235342336. Current Snapshot: MvccSnapshot[applied={T|T < 7333382447515525120}]
I20260925 22:49:34.122087 6089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:34.122267 6089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:34.124874 6089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Sending DeleteTablet for 3 replicas of tablet 6b56d5a1b0c4407087d9fa1233ca1e2c
I20260925 22:49:34.125728 6225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 22:49:34.125805 6366 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:54310
I20260925 22:49:34.125943 6506 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b56d5a1b0c4407087d9fa1233ca1e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:60918
I20260925 22:49:34.126592 6623 tablet_replica.cc:330] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: stopping tablet replica
I20260925 22:49:34.126899 6623 raft_consensus.cc:2229] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:34.127285 6621 tablet_replica.cc:330] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: stopping tablet replica
I20260925 22:49:34.127291 6623 raft_consensus.cc:2258] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.127373 6089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:49:34.127579 6621 raft_consensus.cc:2229] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:34.127712 6622 tablet_replica.cc:330] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: stopping tablet replica
W20260925 22:49:34.127807 6089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:34.127977 6622 raft_consensus.cc:2229] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:34.128098 6621 raft_consensus.cc:2258] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.129518 6623 ts_tablet_manager.cc:1905] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.129518 6621 ts_tablet_manager.cc:1905] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.129774 6622 raft_consensus.cc:2258] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.132393 6621 ts_tablet_manager.cc:1918] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:34.132393 6623 ts_tablet_manager.cc:1918] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:34.132488 6623 log.cc:1196] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/6b56d5a1b0c4407087d9fa1233ca1e2c
I20260925 22:49:34.132488 6621 log.cc:1196] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/6b56d5a1b0c4407087d9fa1233ca1e2c
I20260925 22:49:34.132848 6621 ts_tablet_manager.cc:1939] T 6b56d5a1b0c4407087d9fa1233ca1e2c P c17ef7fb33d64daea19a9b368e0b3f18: Deleting consensus metadata
I20260925 22:49:34.132896 6623 ts_tablet_manager.cc:1939] T 6b56d5a1b0c4407087d9fa1233ca1e2c P e738df10ca1b49479db4d0c4aadbb911: Deleting consensus metadata
I20260925 22:49:34.133754 6225 tablet_service.cc:1467] Processing CreateTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 (DEFAULT_TABLE table=test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:34.133867 6366 tablet_service.cc:1467] Processing CreateTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 (DEFAULT_TABLE table=test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:34.133988 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010b7f8b69dd4cc6afa14460fbecba81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.134080 6074 catalog_manager.cc:4841] TS c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497): tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (table test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]) successfully deleted
I20260925 22:49:34.134155 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010b7f8b69dd4cc6afa14460fbecba81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.135964 6596 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Bootstrap starting.
I20260925 22:49:34.136170 6506 tablet_service.cc:1467] Processing CreateTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 (DEFAULT_TABLE table=test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:34.136404 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010b7f8b69dd4cc6afa14460fbecba81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.136950 6596 tablet_bootstrap.cc:654] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.137576 6595 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Bootstrap starting.
I20260925 22:49:34.138063 6622 ts_tablet_manager.cc:1905] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.138454 6595 tablet_bootstrap.cc:654] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.138707 6597 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Bootstrap starting.
I20260925 22:49:34.139714 6597 tablet_bootstrap.cc:654] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.140481 6595 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: No bootstrap required, opened a new log
I20260925 22:49:34.140599 6595 ts_tablet_manager.cc:1397] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 22:49:34.140805 6622 ts_tablet_manager.cc:1918] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:34.140945 6622 log.cc:1196] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/6b56d5a1b0c4407087d9fa1233ca1e2c
I20260925 22:49:34.141036 6595 raft_consensus.cc:348] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.141168 6595 raft_consensus.cc:374] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.141207 6595 raft_consensus.cc:729] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17ef7fb33d64daea19a9b368e0b3f18, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.141327 6595 consensus_queue.cc:260] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.141484 6622 ts_tablet_manager.cc:1939] T 6b56d5a1b0c4407087d9fa1233ca1e2c P f92a7b5baef442628cf4173c7af62baf: Deleting consensus metadata
I20260925 22:49:34.141579 6595 ts_tablet_manager.cc:1428] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:34.141732 6073 catalog_manager.cc:4841] TS e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131:37933): tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (table test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]) successfully deleted
I20260925 22:49:34.142563 6597 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: No bootstrap required, opened a new log
I20260925 22:49:34.142652 6597 ts_tablet_manager.cc:1397] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:34.143131 6597 raft_consensus.cc:348] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.143249 6597 raft_consensus.cc:374] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.143285 6597 raft_consensus.cc:729] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.143424 6597 consensus_queue.cc:260] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.143680 6597 ts_tablet_manager.cc:1428] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:34.144017 6075 catalog_manager.cc:4841] TS f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643): tablet 6b56d5a1b0c4407087d9fa1233ca1e2c (table test-table [id=8a87326aeb95411baa1a2f0bd3a7d033]) successfully deleted
I20260925 22:49:34.145030 6596 tablet_bootstrap.cc:492] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: No bootstrap required, opened a new log
I20260925 22:49:34.145114 6596 ts_tablet_manager.cc:1397] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 22:49:34.145553 6596 raft_consensus.cc:348] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.145980 6596 raft_consensus.cc:374] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.146020 6596 raft_consensus.cc:729] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92a7b5baef442628cf4173c7af62baf, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.146130 6596 consensus_queue.cc:260] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.146387 6596 ts_tablet_manager.cc:1428] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:49:34.337723 6618 raft_consensus.cc:482] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:34.337853 6618 raft_consensus.cc:504] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.338356 6618 leader_election.cc:288] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.338781 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010b7f8b69dd4cc6afa14460fbecba81" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf" is_pre_election: true
I20260925 22:49:34.338793 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010b7f8b69dd4cc6afa14460fbecba81" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" is_pre_election: true
I20260925 22:49:34.338943 6381 raft_consensus.cc:2454] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.338953 6246 raft_consensus.cc:2454] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.339241 6458 leader_election.cc:302] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17ef7fb33d64daea19a9b368e0b3f18, e738df10ca1b49479db4d0c4aadbb911; no voters:
I20260925 22:49:34.339418 6618 raft_consensus.cc:2781] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:34.339514 6618 raft_consensus.cc:482] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:34.339569 6618 raft_consensus.cc:3037] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.340634 6618 raft_consensus.cc:504] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.340992 6618 leader_election.cc:288] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Requested vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.341344 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010b7f8b69dd4cc6afa14460fbecba81" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
I20260925 22:49:34.341342 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010b7f8b69dd4cc6afa14460fbecba81" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf"
I20260925 22:49:34.341475 6381 raft_consensus.cc:3037] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.341477 6246 raft_consensus.cc:3037] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.342309 6381 raft_consensus.cc:2454] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 1.
I20260925 22:49:34.342593 6459 leader_election.cc:302] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e738df10ca1b49479db4d0c4aadbb911, f92a7b5baef442628cf4173c7af62baf; no voters:
I20260925 22:49:34.342759 6618 raft_consensus.cc:2781] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:34.342887 6618 raft_consensus.cc:686] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 1 LEADER]: Becoming Leader. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Running, Role: LEADER
I20260925 22:49:34.342962 6246 raft_consensus.cc:2454] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 1.
I20260925 22:49:34.343035 6618 consensus_queue.cc:237] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.344734 6088 catalog_manager.cc:5495] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 reported cstate change: term changed from 0 to 1, leader changed from <none> to e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } health_report { overall_health: HEALTHY } } }
Sep 25 22:49:34 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376572, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:34.414733 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.406283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:38009 (local address 127.0.0.1:38036)
0925 22:49:34.406629 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.406643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.406751 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.407031 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.407034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.407423 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:34.407515 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.407521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.408280 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.408283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.408863 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.408878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.409297 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.409299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.409359 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.409892 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:34.409903 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:34.411377 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:34.413215 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:34.413223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:34.413228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:34.413474 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:34.413818 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:34.413820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:34.413822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:34.413914 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:34.414259 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:34.414264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:34.414361 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.414453 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.414586 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":125,"threads_started":1}
I20260925 22:49:34.414954 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.406361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.4.219.190:38009)
0925 22:49:34.406620 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.406632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.406717 (+ 85us) server_negotiation.cc:407] Connection header received
0925 22:49:34.406847 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.406849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.406889 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.406980 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:34.407655 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.408184 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.408983 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.409195 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.409272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.411582 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:34.411585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:34.411587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:34.411606 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:34.413106 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.413589 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.413595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.413597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.413730 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.413988 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.413990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.413991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.414170 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:34.414248 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.414555 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.414699 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260925 22:49:34.424015 6088 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:38036:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:49:34.425016 6088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:34.430065 6225 tablet_service.cc:1467] Processing CreateTablet for tablet 5622aa9283284eb8a9f7cf4aeb5d484f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5f6d690a04937bb93094156967411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:34.430508 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5622aa9283284eb8a9f7cf4aeb5d484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.432906 6366 tablet_service.cc:1467] Processing CreateTablet for tablet 5622aa9283284eb8a9f7cf4aeb5d484f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5f6d690a04937bb93094156967411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:34.433075 6595 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: Bootstrap starting.
I20260925 22:49:34.433283 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5622aa9283284eb8a9f7cf4aeb5d484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.433919 6595 tablet_bootstrap.cc:654] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.434927 6595 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: No bootstrap required, opened a new log
I20260925 22:49:34.434985 6595 ts_tablet_manager.cc:1397] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 22:49:34.435319 6595 raft_consensus.cc:348] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.435410 6596 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: Bootstrap starting.
I20260925 22:49:34.435436 6595 raft_consensus.cc:374] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.435472 6595 raft_consensus.cc:729] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c17ef7fb33d64daea19a9b368e0b3f18, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.435603 6595 consensus_queue.cc:260] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.435758 6506 tablet_service.cc:1467] Processing CreateTablet for tablet 5622aa9283284eb8a9f7cf4aeb5d484f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5f6d690a04937bb93094156967411]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:34.435935 6595 ts_tablet_manager.cc:1428] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:34.435869 6603 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.436048 6603 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.436177 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5622aa9283284eb8a9f7cf4aeb5d484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:34.436303 6596 tablet_bootstrap.cc:654] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.437923 6596 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: No bootstrap required, opened a new log
I20260925 22:49:34.438007 6596 ts_tablet_manager.cc:1397] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:34.438469 6596 raft_consensus.cc:348] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.438575 6596 raft_consensus.cc:374] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.438598 6596 raft_consensus.cc:729] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f92a7b5baef442628cf4173c7af62baf, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.438741 6596 consensus_queue.cc:260] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.438844 6597 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: Bootstrap starting.
I20260925 22:49:34.438997 6596 ts_tablet_manager.cc:1428] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:34.439168 6602 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.439298 6602 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.439685 6597 tablet_bootstrap.cc:654] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:34.440795 6597 tablet_bootstrap.cc:492] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: No bootstrap required, opened a new log
I20260925 22:49:34.440898 6597 ts_tablet_manager.cc:1397] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:34.441279 6597 raft_consensus.cc:348] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.441390 6597 raft_consensus.cc:374] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:34.441432 6597 raft_consensus.cc:729] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Initialized, Role: FOLLOWER
I20260925 22:49:34.441587 6597 consensus_queue.cc:260] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.441874 6597 ts_tablet_manager.cc:1428] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:49:34.441843 6618 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.441975 6618 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.479895 6618 raft_consensus.cc:482] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:34.480037 6618 raft_consensus.cc:504] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.480525 6618 leader_election.cc:288] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.481012 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5622aa9283284eb8a9f7cf4aeb5d484f" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" is_pre_election: true
I20260925 22:49:34.481031 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5622aa9283284eb8a9f7cf4aeb5d484f" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf" is_pre_election: true
I20260925 22:49:34.481168 6246 raft_consensus.cc:2454] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.481205 6381 raft_consensus.cc:2454] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 0.
I20260925 22:49:34.481490 6458 leader_election.cc:302] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17ef7fb33d64daea19a9b368e0b3f18, e738df10ca1b49479db4d0c4aadbb911; no voters:
I20260925 22:49:34.481719 6618 raft_consensus.cc:2781] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:34.481801 6618 raft_consensus.cc:482] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:34.481855 6618 raft_consensus.cc:3037] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.482874 6618 raft_consensus.cc:504] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.483227 6618 leader_election.cc:288] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Requested vote from peers c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497), f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643)
I20260925 22:49:34.483590 6246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5622aa9283284eb8a9f7cf4aeb5d484f" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c17ef7fb33d64daea19a9b368e0b3f18"
I20260925 22:49:34.483654 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5622aa9283284eb8a9f7cf4aeb5d484f" candidate_uuid: "e738df10ca1b49479db4d0c4aadbb911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f92a7b5baef442628cf4173c7af62baf"
I20260925 22:49:34.483716 6246 raft_consensus.cc:3037] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.483755 6381 raft_consensus.cc:3037] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:34.484961 6246 raft_consensus.cc:2454] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 1.
I20260925 22:49:34.485297 6458 leader_election.cc:302] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c17ef7fb33d64daea19a9b368e0b3f18, e738df10ca1b49479db4d0c4aadbb911; no voters:
I20260925 22:49:34.485503 6618 raft_consensus.cc:2781] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:34.485589 6618 raft_consensus.cc:686] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [term 1 LEADER]: Becoming Leader. State: Replica: e738df10ca1b49479db4d0c4aadbb911, State: Running, Role: LEADER
I20260925 22:49:34.485582 6381 raft_consensus.cc:2454] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e738df10ca1b49479db4d0c4aadbb911 in term 1.
I20260925 22:49:34.485735 6618 consensus_queue.cc:237] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } }
I20260925 22:49:34.486447 6618 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: TxnStatusTablet state changed. Reason: New leader e738df10ca1b49479db4d0c4aadbb911. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.486550 6618 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:34.486825 6639 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:49:34.487056 6639 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:49:34.487596 6089 catalog_manager.cc:5495] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 reported cstate change: term changed from 0 to 1, leader changed from <none> to e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:34.509503 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.505457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:37933 (local address 127.0.0.1:60938)
0925 22:49:34.505652 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.505666 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.505760 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.506336 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.506339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.506346 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:34.506450 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.506484 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.507524 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.507526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.508121 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.508128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.508718 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.508721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.508792 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.508838 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:34.509201 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:34.509216 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.509290 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.509361 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 22:49:34.509516 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.505936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.4.219.131:37933)
0925 22:49:34.506084 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.506089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.506105 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:49:34.506156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.506160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.506211 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.506291 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:34.506596 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.507387 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.508235 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.508606 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.508709 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.508969 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:34.509055 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:34.509125 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.509316 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.509371 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:49:34.512131 6246 raft_consensus.cc:1261] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.512144 6381 raft_consensus.cc:1261] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.512598 6608 consensus_queue.cc:1035] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.512728 6618 consensus_queue.cc:1035] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.514165 6602 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: TxnStatusTablet state changed. Reason: New leader e738df10ca1b49479db4d0c4aadbb911. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.514278 6602 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.514215 6603 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: TxnStatusTablet state changed. Reason: New leader e738df10ca1b49479db4d0c4aadbb911. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.514336 6603 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.516810 6618 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.516920 6618 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:34.517040 6618 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.517110 6618 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P e738df10ca1b49479db4d0c4aadbb911: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:34.517365 6602 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.517462 6602 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P f92a7b5baef442628cf4173c7af62baf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.518779 6603 tablet_replica.cc:439] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e738df10ca1b49479db4d0c4aadbb911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 } } peers { permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 } } peers { permanent_uuid: "e738df10ca1b49479db4d0c4aadbb911" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 37933 } } }
I20260925 22:49:34.518889 6603 tablet_replica.cc:442] T 5622aa9283284eb8a9f7cf4aeb5d484f P c17ef7fb33d64daea19a9b368e0b3f18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:34.532802 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.529295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44771 (local address 127.4.219.131:37933)
0925 22:49:34.529416 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.529421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.529554 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:49:34.529681 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.529683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.529724 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.529826 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:34.530041 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.530538 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.531767 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.532001 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.532080 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.532373 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:34.532435 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:34.532487 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.532634 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.532678 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:34.532861 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.529245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:37933 (local address 127.4.219.131:44771)
0925 22:49:34.529486 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:34.529501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:34.529605 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:34.529837 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:34.529839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:34.529847 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:34.529960 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.529967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.530642 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.530645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.531637 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:34.531648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.532108 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.532111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:34.532196 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.532247 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:34.532514 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:34.532519 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:49:34.532625 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:49:34.532705 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260925 22:49:34.538679 6246 raft_consensus.cc:1261] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.538692 6381 raft_consensus.cc:1261] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Refusing update from remote peer e738df10ca1b49479db4d0c4aadbb911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:34.539088 6608 consensus_queue.cc:1035] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f92a7b5baef442628cf4173c7af62baf" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.539228 6601 consensus_queue.cc:1035] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c17ef7fb33d64daea19a9b368e0b3f18" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:34.575392 6089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:34.575555 6089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:34.577584 6089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90e574a4d5b544e483486695375b0537: Sending DeleteTablet for 3 replicas of tablet 010b7f8b69dd4cc6afa14460fbecba81
I20260925 22:49:34.578054 6225 tablet_service.cc:1514] Processing DeleteTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 22:49:34.578285 6621 tablet_replica.cc:330] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: stopping tablet replica
I20260925 22:49:34.578265 6366 tablet_service.cc:1514] Processing DeleteTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:54310
I20260925 22:49:34.578398 6621 raft_consensus.cc:2229] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:34.578424 6622 tablet_replica.cc:330] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: stopping tablet replica
I20260925 22:49:34.578442 6505 tablet_service.cc:1514] Processing DeleteTablet for tablet 010b7f8b69dd4cc6afa14460fbecba81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:34 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:60918
I20260925 22:49:34.578539 6621 raft_consensus.cc:2258] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.578548 6622 raft_consensus.cc:2229] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:34.578583 6623 tablet_replica.cc:330] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: stopping tablet replica
I20260925 22:49:34.578650 6622 raft_consensus.cc:2258] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.578677 6623 raft_consensus.cc:2229] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:34.578855 6623 raft_consensus.cc:2258] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:34.579471 6621 ts_tablet_manager.cc:1905] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.579918 6623 ts_tablet_manager.cc:1905] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.582278 6621 ts_tablet_manager.cc:1918] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:34.582343 6621 log.cc:1196] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/010b7f8b69dd4cc6afa14460fbecba81
I20260925 22:49:34.582777 6621 ts_tablet_manager.cc:1939] T 010b7f8b69dd4cc6afa14460fbecba81 P c17ef7fb33d64daea19a9b368e0b3f18: Deleting consensus metadata
I20260925 22:49:34.583510 6074 catalog_manager.cc:4841] TS c17ef7fb33d64daea19a9b368e0b3f18 (127.4.219.129:34497): tablet 010b7f8b69dd4cc6afa14460fbecba81 (table test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]) successfully deleted
I20260925 22:49:34.583748 6623 ts_tablet_manager.cc:1918] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:34.583825 6623 log.cc:1196] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/010b7f8b69dd4cc6afa14460fbecba81
I20260925 22:49:34.584082 6623 ts_tablet_manager.cc:1939] T 010b7f8b69dd4cc6afa14460fbecba81 P e738df10ca1b49479db4d0c4aadbb911: Deleting consensus metadata
I20260925 22:49:34.584723 6073 catalog_manager.cc:4841] TS e738df10ca1b49479db4d0c4aadbb911 (127.4.219.131:37933): tablet 010b7f8b69dd4cc6afa14460fbecba81 (table test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]) successfully deleted
I20260925 22:49:34.585129 6622 ts_tablet_manager.cc:1905] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:34.587779 6622 ts_tablet_manager.cc:1918] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:34.587863 6622 log.cc:1196] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/010b7f8b69dd4cc6afa14460fbecba81
I20260925 22:49:34.588119 6622 ts_tablet_manager.cc:1939] T 010b7f8b69dd4cc6afa14460fbecba81 P f92a7b5baef442628cf4173c7af62baf: Deleting consensus metadata
I20260925 22:49:34.588857 6075 catalog_manager.cc:4841] TS f92a7b5baef442628cf4173c7af62baf (127.4.219.130:34643): tablet 010b7f8b69dd4cc6afa14460fbecba81 (table test-table [id=4f1f3fdd9d8c49299e34bb9329bae2ef]) successfully deleted
Sep 25 22:49:34 dist-test-slave-kb1g krb5kdc[6044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376573, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:34.600597 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.589154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.4.219.129:34497)
0925 22:49:34.589418 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.589423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.589611 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:49:34.589739 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.589742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.589788 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.589880 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:34.590711 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.591529 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.592746 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.592982 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.593068 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.596811 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:34.596824 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:34.596839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:34.596866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:34.598647 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.599192 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.599199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.599218 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.599319 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.599608 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.599613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.599615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.599874 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:34.600103 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.600272 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.600400 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260925 22:49:34.601079 6286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57136
I20260925 22:49:34.611397 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:34.602982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.4.219.190:38009)
0925 22:49:34.603133 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:34.603138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:34.603220 (+ 82us) server_negotiation.cc:407] Connection header received
0925 22:49:34.603373 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:34.603376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:34.603420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:34.603502 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:34.604295 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.604983 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.605869 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:34.606139 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:34.606265 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:34.607788 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:34.607791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:34.607794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:34.607820 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:34.609557 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.610045 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.610054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.610056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.610142 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:34.610442 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:34.610446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:34.610448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:34.610647 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:34.610801 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:34.611064 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 22:49:34.611209 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 22:49:34.612036 6089 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38052
I20260925 22:49:34.613344 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6161
I20260925 22:49:34.621963 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6301
I20260925 22:49:34.628664 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6441
I20260925 22:49:34.637238 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6057
2026-09-25T22:49:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3135 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:34 dist-test-slave-kb1g krb5kdc[6658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:34 dist-test-slave-kb1g krb5kdc[6658](info): set up 2 sockets
Sep 25 22:49:34 dist-test-slave-kb1g krb5kdc[6658](info): commencing operation
krb5kdc: starting...
W20260925 22:49:35.668304 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:35 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:35.788020 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:46725
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:35.897924 6671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:35.898187 6671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:35.898227 6671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:35.902098 6671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:35.902144 6671 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:49:35.902161 6671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:35.902179 6671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:35.902195 6671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:35.902210 6671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:35.907271 6671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:46725
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:35.908227 6671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:35.909166 6671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:35.917282 6676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:35.917291 6677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:35.917339 6671 server_base.cc:1034] running on GCE node
W20260925 22:49:35.917296 6679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:35.918030 6671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:35.919114 6671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:35.920275 6671 hybrid_clock.cc:648] HybridClock initialized: now 1790376575920234 us; error 51 us; skew 500 ppm
Sep 25 22:49:35 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:35.923069 6671 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:35.924042 6671 webserver.cc:460] Webserver started at http://127.4.219.190:38747/ using document root <none> and password file <none>
I20260925 22:49:35.924628 6671 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:35.924707 6671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:35.924875 6671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:35.926828 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "5cf9de2c9cb24a5fb638581f7a575864"
format_stamp: "Formatted at 2026-09-25 22:49:35 on dist-test-slave-kb1g"
server_key: "ee075d9cb79d958434e17e568255fe84"
server_key_iv: "6664c802bb1a2282078def142be93e26"
server_key_version: "encryptionkey@0"
I20260925 22:49:35.927343 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "5cf9de2c9cb24a5fb638581f7a575864"
format_stamp: "Formatted at 2026-09-25 22:49:35 on dist-test-slave-kb1g"
server_key: "ee075d9cb79d958434e17e568255fe84"
server_key_iv: "6664c802bb1a2282078def142be93e26"
server_key_version: "encryptionkey@0"
I20260925 22:49:35.930985 6671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:49:35.933421 6686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:35.934687 6671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:35.934844 6671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "5cf9de2c9cb24a5fb638581f7a575864"
format_stamp: "Formatted at 2026-09-25 22:49:35 on dist-test-slave-kb1g"
server_key: "ee075d9cb79d958434e17e568255fe84"
server_key_iv: "6664c802bb1a2282078def142be93e26"
server_key_version: "encryptionkey@0"
I20260925 22:49:35.934975 6671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:35.946511 6671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:35.949640 6671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:35.949832 6671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:35.957778 6671 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:46725
I20260925 22:49:35.957799 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:46725 every 8 connection(s)
I20260925 22:49:35.958715 6671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:35.961436 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:35.964020 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6671
I20260925 22:49:35.964133 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:35.964422 4974 external_mini_cluster.cc:1428] Setting key c42d77b69db7bfae1ecb547ca87fd4ae
I20260925 22:49:35.967337 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Bootstrap starting.
I20260925 22:49:35.969879 6749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:35.970716 6749 log.cc:824] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:35.973178 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: No bootstrap required, opened a new log
Sep 25 22:49:35 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:35.977801 6749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } }
I20260925 22:49:35.978091 6749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:35.978157 6749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf9de2c9cb24a5fb638581f7a575864, State: Initialized, Role: FOLLOWER
I20260925 22:49:35.978634 6749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } }
I20260925 22:49:35.978796 6749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:35.978876 6749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:35.978971 6749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:35.979456 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:35.965798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.4.219.190:46725)
0925 22:49:35.966186 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:35.966202 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:35.966237 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:49:35.966839 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:35.966855 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:35.967182 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:35.967486 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:35.968312 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:35.969363 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:35.970544 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:35.971117 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:35.971304 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:35.974131 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:35.974138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:35.974153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:35.974183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:35.976087 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:35.976646 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:35.976655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:35.976661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:35.976847 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:35.977188 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:35.977191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:35.977193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:35.977691 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:35.977914 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:35.978117 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:49:35.978431 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260925 22:49:35.980022 6749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } }
I20260925 22:49:35.980365 6749 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf9de2c9cb24a5fb638581f7a575864; no voters:
I20260925 22:49:35.980748 6749 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:35.980819 6754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:35.981036 6754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [term 1 LEADER]: Becoming Leader. State: Replica: 5cf9de2c9cb24a5fb638581f7a575864, State: Running, Role: LEADER
I20260925 22:49:35.981348 6754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } }
I20260925 22:49:35.982046 6749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:35.982511 6754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf9de2c9cb24a5fb638581f7a575864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } } }
I20260925 22:49:35.982625 6754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:35.983006 6755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf9de2c9cb24a5fb638581f7a575864. Latest consensus state: current_term: 1 leader_uuid: "5cf9de2c9cb24a5fb638581f7a575864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf9de2c9cb24a5fb638581f7a575864" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46725 } } }
I20260925 22:49:35.983176 6755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:35.984277 6758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:35.987330 6758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:35.992880 6758 catalog_manager.cc:1347] Generated new cluster ID: d636d65c21be48b3ae8d9f4aff0b6b0e
I20260925 22:49:35.992945 6758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:36.014876 6758 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:36.015827 6758 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:36.023598 6758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Generated new TSK 0
I20260925 22:49:36.024283 6758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:36.060213 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46725
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:36.171680 6775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:36.171975 6775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:36.172062 6775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:36.176245 6775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:36.176327 6775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:36.176394 6775 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:49:36.176443 6775 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:36.176512 6775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:36.181936 6775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:36.183107 6775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:36.184295 6775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:36.193815 6783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:36.193782 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:36.193840 6781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:36.193982 6775 server_base.cc:1034] running on GCE node
I20260925 22:49:36.194476 6775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:36.195151 6775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:36.196381 6775 hybrid_clock.cc:648] HybridClock initialized: now 1790376576196354 us; error 38 us; skew 500 ppm
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:36.199147 6775 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:36.200028 6775 webserver.cc:460] Webserver started at http://127.4.219.129:39713/ using document root <none> and password file <none>
I20260925 22:49:36.200642 6775 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:36.200728 6775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:36.200963 6775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:36.202975 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "14dbc5a27ade4518a15aec870df212c6"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "37ec3c7f3488a497218707e4606a4cda"
server_key_iv: "f6959824884be6208acb71730f8a50a9"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.203552 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "14dbc5a27ade4518a15aec870df212c6"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "37ec3c7f3488a497218707e4606a4cda"
server_key_iv: "f6959824884be6208acb71730f8a50a9"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.207353 6775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:49:36.209962 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.211180 6775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:36.211316 6775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "14dbc5a27ade4518a15aec870df212c6"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "37ec3c7f3488a497218707e4606a4cda"
server_key_iv: "f6959824884be6208acb71730f8a50a9"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.211431 6775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:36.246873 6775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:36.250082 6775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:36.250291 6775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:36.251739 6775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:36.251811 6775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.251883 6775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:36.251924 6775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:36.262607 6775 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:34381
I20260925 22:49:36.262679 6905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:34381 every 8 connection(s)
I20260925 22:49:36.263695 6775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:36.266283 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.252869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:54337 (local address 127.4.219.190:46725)
0925 22:49:36.253012 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.253017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.253970 (+ 953us) server_negotiation.cc:407] Connection header received
0925 22:49:36.255404 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.255409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.255475 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.255572 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.256899 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.257438 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.258783 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.259035 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.259152 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.262423 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.262427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.262430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.262459 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.264046 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.264691 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.264696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.264698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.264786 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.265147 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.265150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.265152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.265340 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.265474 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.265985 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.266112 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:49:36.267021 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6775
I20260925 22:49:36.267141 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:36.267194 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.253285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.129:54337)
0925 22:49:36.253799 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.253855 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.255150 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.255749 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.255755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.256082 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.256748 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.256764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.257574 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.257577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.258665 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.258674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.259198 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.259202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.259374 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.260520 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.260589 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.262248 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.264162 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.264168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.264182 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.264579 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.264890 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.264894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.264895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.265045 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.265483 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.265492 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.265730 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.265924 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.266235 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":103,"threads_started":1}
I20260925 22:49:36.267452 4974 external_mini_cluster.cc:1428] Setting key 1dc616551ea28ebd0bad2dce4a4066f0
I20260925 22:49:36.268641 6906 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46725
I20260925 22:49:36.268774 6906 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:36.269322 6906 heartbeater.cc:511] Master 127.4.219.190:46725 requested a full tablet report, sending...
I20260925 22:49:36.270933 6702 ts_manager.cc:194] Registered new tserver with Master: 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381)
I20260925 22:49:36.272222 6702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:54337
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:36.280870 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.272767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43491 (local address 127.4.219.190:46725)
0925 22:49:36.272885 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.272891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.273279 (+ 388us) server_negotiation.cc:407] Connection header received
0925 22:49:36.273474 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.273480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.273543 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.273634 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.274123 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.275044 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.275845 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.276220 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.276342 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.277536 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.277543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.277546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.277585 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.278984 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.279576 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.279579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.279580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.279669 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.279942 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.279946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.279947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.280208 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.280314 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.280641 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.280745 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:36.281055 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.272925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.129:43491)
0925 22:49:36.273200 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.273214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.273356 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.273667 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.273670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.273877 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.273987 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.273995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.275181 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.275183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.275730 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.275737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.276344 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.276347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.276417 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.277049 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.277068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.277390 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.279215 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.279219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.279222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.279466 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.279762 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.279784 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.279789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.279874 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.280311 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.280317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.280421 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.280585 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.280765 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:36.296643 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46725
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:36.408715 6915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:36.408985 6915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:36.409070 6915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:36.413046 6915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:36.413129 6915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:36.413194 6915 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:49:36.413235 6915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:36.413306 6915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:36.421241 6915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:36.422706 6915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:36.423831 6915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:36.434118 6915 server_base.cc:1034] running on GCE node
W20260925 22:49:36.435127 6920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:36.435276 6923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:36.435691 6921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:36.436122 6915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:36.436857 6915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:36.438041 6915 hybrid_clock.cc:648] HybridClock initialized: now 1790376576438019 us; error 30 us; skew 500 ppm
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:36.441004 6915 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:36.441856 6915 webserver.cc:460] Webserver started at http://127.4.219.130:45485/ using document root <none> and password file <none>
I20260925 22:49:36.442457 6915 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:36.442550 6915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:36.442776 6915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:36.444860 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "c707d6246e844828a8a581fc15228167"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "40f98b48f49302ea2458a7f89892188b"
server_key_iv: "8841698d57dd851150ebfe95b4a68257"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.445410 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "c707d6246e844828a8a581fc15228167"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "40f98b48f49302ea2458a7f89892188b"
server_key_iv: "8841698d57dd851150ebfe95b4a68257"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.449007 6915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:36.451363 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.452661 6915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:36.452796 6915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "c707d6246e844828a8a581fc15228167"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "40f98b48f49302ea2458a7f89892188b"
server_key_iv: "8841698d57dd851150ebfe95b4a68257"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.452904 6915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:36.468626 6915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:36.472234 6915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:36.472430 6915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:36.473892 6915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:36.473965 6915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.474023 6915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:36.474074 6915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:36.485008 6915 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40507
I20260925 22:49:36.485072 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40507 every 8 connection(s)
I20260925 22:49:36.486203 6915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:36.488835 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.474767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:55363 (local address 127.4.219.190:46725)
0925 22:49:36.474889 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.474892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.475978 (+ 1086us) server_negotiation.cc:407] Connection header received
0925 22:49:36.477143 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.477147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.477207 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.477312 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.478873 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.479725 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.481204 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.481451 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.481569 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.485096 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.485100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.485103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.485133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.486843 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.487374 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.487377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.487378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.487468 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.487782 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.487785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.487786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.488001 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.488083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.488584 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.488708 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:36.489650 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.475334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.130:55363)
0925 22:49:36.475813 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.475855 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.476886 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.477511 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.477520 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.477996 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.478691 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.478714 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.479859 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.479867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.481079 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.481089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.481629 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.481636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.481861 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.482482 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.482506 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.484887 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.487000 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.487006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.487021 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.487275 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.487554 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.487556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.487558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.487690 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.488091 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.488096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.488312 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.488456 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.488788 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":128,"threads_started":1}
I20260925 22:49:36.490901 7046 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46725
I20260925 22:49:36.491050 7046 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:36.491621 7046 heartbeater.cc:511] Master 127.4.219.190:46725 requested a full tablet report, sending...
I20260925 22:49:36.492784 6702 ts_manager.cc:194] Registered new tserver with Master: c707d6246e844828a8a581fc15228167 (127.4.219.130:40507)
I20260925 22:49:36.493111 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 6915
I20260925 22:49:36.493238 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:36.493564 6702 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:55363
I20260925 22:49:36.493619 4974 external_mini_cluster.cc:1428] Setting key 6ad3a162deb928c00e728dd2b2b832a1
I20260925 22:49:36.501315 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.493585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:58251 (local address 127.4.219.190:46725)
0925 22:49:36.493780 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.493786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.494258 (+ 472us) server_negotiation.cc:407] Connection header received
0925 22:49:36.494425 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.494429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.494487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.494578 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.495093 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.495654 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.496692 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.496900 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.496980 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.498237 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.498240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.498242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.498262 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.499647 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.500070 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.500074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.500076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.500153 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.500425 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.500428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.500429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.500707 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.500809 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.501075 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.501179 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 22:49:36.501327 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.493490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.130:58251)
0925 22:49:36.494161 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.494176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.494315 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.494595 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.494598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.494832 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.494976 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.494983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.495787 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.495790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.496540 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.496582 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.497019 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.497025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.497157 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.497691 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.497711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.498117 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.499756 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.499761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.499763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.499960 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.500240 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.500245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.500247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.500331 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.500809 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.500812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.500905 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.501013 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.501150 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:36.523145 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46725
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:36.634589 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:36.634832 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:36.634877 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:36.639037 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:36.639092 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:36.639132 7055 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:49:36.639146 7055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:49:36.639209 7055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:36.644363 7055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:46725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:36.645406 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:36.646399 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:36.655854 7060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:36.655861 7061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:36.655938 7055 server_base.cc:1034] running on GCE node
W20260925 22:49:36.655854 7063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:36.656497 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:36.657240 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:36.658440 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790376576658414 us; error 42 us; skew 500 ppm
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:36.661209 7055 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:36.662143 7055 webserver.cc:460] Webserver started at http://127.4.219.131:33055/ using document root <none> and password file <none>
I20260925 22:49:36.662706 7055 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:36.662796 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:36.663035 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:36.665091 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "6ec1eb6139e34cb086a0982b776a60e9"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "365a8beac2cdb1b3238b67bc98054215"
server_key_iv: "52804718369ee24a2bd33754c798e7e2"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.665650 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "6ec1eb6139e34cb086a0982b776a60e9"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "365a8beac2cdb1b3238b67bc98054215"
server_key_iv: "52804718369ee24a2bd33754c798e7e2"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.669477 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:49:36.672024 7070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.673324 7055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:36.673482 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "6ec1eb6139e34cb086a0982b776a60e9"
format_stamp: "Formatted at 2026-09-25 22:49:36 on dist-test-slave-kb1g"
server_key: "365a8beac2cdb1b3238b67bc98054215"
server_key_iv: "52804718369ee24a2bd33754c798e7e2"
server_key_version: "encryptionkey@0"
I20260925 22:49:36.673597 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:36.690065 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:36.693866 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:36.694214 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:36.695645 7055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:36.695717 7055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:36.695789 7055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:36.695842 7055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:36.706799 7055 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:33859
I20260925 22:49:36.706881 7185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:33859 every 8 connection(s)
I20260925 22:49:36.707860 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:36.709524 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.696649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:46569 (local address 127.4.219.190:46725)
0925 22:49:36.696787 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.696790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.697821 (+ 1031us) server_negotiation.cc:407] Connection header received
0925 22:49:36.699038 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.699041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.699098 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.699211 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.700723 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.701293 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.702558 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.702838 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.702960 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.705609 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.705615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.705619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.705651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.707369 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.707864 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.707869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.707874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.707968 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.708326 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.708330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.708331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.708599 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.708704 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.709279 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.709376 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:36.709774 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7055
I20260925 22:49:36.709887 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:36.710120 4974 external_mini_cluster.cc:1428] Setting key 1c70a1c0e8e79b9909a14d96b22f683f
I20260925 22:49:36.710456 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.697193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.131:46569)
0925 22:49:36.697667 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.697707 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.698753 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.699410 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.699421 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.699799 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.700566 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.700589 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.701441 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.701444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.702435 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.702444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.703009 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.703013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.703170 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.703843 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.703858 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.705418 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.707477 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.707481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.707492 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.707738 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.708078 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.708080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.708082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.708215 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.708752 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.708758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.709052 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.709195 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.709483 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260925 22:49:36.711745 7186 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46725
I20260925 22:49:36.711863 7186 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:36.712363 7186 heartbeater.cc:511] Master 127.4.219.190:46725 requested a full tablet report, sending...
I20260925 22:49:36.713370 6703 ts_manager.cc:194] Registered new tserver with Master: 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:36.714066 6703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:46569
I20260925 22:49:36.720508 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.713925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:47563 (local address 127.4.219.190:46725)
0925 22:49:36.714061 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.714066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.714269 (+ 203us) server_negotiation.cc:407] Connection header received
0925 22:49:36.714421 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.714424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.714511 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.714586 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.715079 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.715796 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.716737 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.716938 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.717018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.717921 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.717923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.717925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.717945 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.719310 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.719664 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.719666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.719667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.719731 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.719943 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.719945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.719946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.720089 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.720157 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.720332 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.720412 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:36.720572 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.713864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.4.219.131:47563)
0925 22:49:36.714184 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.714200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.714312 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.714693 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.714696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.714870 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.714962 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.714967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.715911 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.715913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.716620 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.716629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.717042 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.717044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.717109 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.717511 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.717521 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.717808 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.719398 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.719400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.719402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.719584 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.719804 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.719806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.719807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.719873 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.720160 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.720162 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.720227 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.720302 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.720398 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260925 22:49:36.723989 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:36.749397 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.740914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.4.219.190:46725)
0925 22:49:36.741045 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.741049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.741062 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:49:36.741104 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.741107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.741147 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.741228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.741980 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.742542 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.743408 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.743624 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.743707 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.746259 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.746263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.746265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.746292 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.747713 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.748132 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.748135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.748136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.748222 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.748501 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.748503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.748505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.748794 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.748915 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.749176 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.749277 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:49:36.752419 6703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:36.755481 6703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:36.770558 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.764336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.0.0.1:57458)
0925 22:49:36.765727 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.765738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.765816 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.766259 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.766261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.766277 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:36.766365 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.766374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767815 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.767819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.769125 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.769135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.770028 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.770136 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770170 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.770287 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.770404 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1335,"mutex_wait_us":560,"thread_start_us":103,"threads_started":1}
I20260925 22:49:36.770558 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.764600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.4.219.129:34381)
0925 22:49:36.765202 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.765221 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.765811 (+ 590us) server_negotiation.cc:407] Connection header received
0925 22:49:36.765924 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.765929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.766107 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.766282 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:36.766477 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767556 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.769248 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.769909 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770039 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770195 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.770316 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.770407 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":80,"threads_started":1}
I20260925 22:49:36.770926 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.764336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33859 (local address 127.0.0.1:40348)
0925 22:49:36.765175 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.765230 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.765363 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.765955 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.765960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.765980 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:36.766119 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.766125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767542 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.767545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.769145 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.769156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.770391 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.770582 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770643 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.770720 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.770801 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":153,"thread_start_us":60,"threads_started":1}
I20260925 22:49:36.771368 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.764511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.4.219.131:33859)
0925 22:49:36.765410 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.765423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.765450 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:49:36.765527 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.765534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.765807 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.765988 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:36.766240 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767436 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.769290 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.770093 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770197 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770334 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.771104 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.771212 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":156,"threads_started":1}
I20260925 22:49:36.771370 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.764592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40507 (local address 127.0.0.1:43560)
0925 22:49:36.765634 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.765648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.765751 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.766392 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.766398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.766416 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:36.766504 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.766509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767440 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.767446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.769168 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.769174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.770391 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.770578 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770972 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.771189 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.771253 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":179,"thread_start_us":80,"threads_started":1}
I20260925 22:49:36.772662 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.765164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.4.219.130:40507)
0925 22:49:36.765586 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.765598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.765747 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:49:36.765913 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.765921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.766127 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.766271 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:36.766610 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.767323 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.769290 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.770093 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.770197 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.770326 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.772403 (+ 2077us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.772501 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":107,"threads_started":1}
I20260925 22:49:36.773850 6838 tablet_service.cc:1467] Processing CreateTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f (DEFAULT_TABLE table=test-table [id=c6d67e3260c14abc995595637f808198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:36.774164 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f (DEFAULT_TABLE table=test-table [id=c6d67e3260c14abc995595637f808198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:36.774932 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c133a5c90c3c431aa42ecb694d7f3c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.775192 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c133a5c90c3c431aa42ecb694d7f3c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.775204 6980 tablet_service.cc:1467] Processing CreateTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f (DEFAULT_TABLE table=test-table [id=c6d67e3260c14abc995595637f808198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:36.776135 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c133a5c90c3c431aa42ecb694d7f3c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.781525 7209 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Bootstrap starting.
I20260925 22:49:36.783835 7209 tablet_bootstrap.cc:654] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.784014 7210 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Bootstrap starting.
I20260925 22:49:36.784473 7211 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Bootstrap starting.
I20260925 22:49:36.785292 7209 log.cc:824] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:36.786235 7210 tablet_bootstrap.cc:654] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.786566 7211 tablet_bootstrap.cc:654] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.787091 7210 log.cc:824] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:36.787302 7211 log.cc:824] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:36.787359 7209 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: No bootstrap required, opened a new log
I20260925 22:49:36.787596 7209 ts_tablet_manager.cc:1397] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:49:36.788982 7210 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: No bootstrap required, opened a new log
I20260925 22:49:36.789177 7210 ts_tablet_manager.cc:1397] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:36.789292 7211 tablet_bootstrap.cc:492] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: No bootstrap required, opened a new log
I20260925 22:49:36.789459 7211 ts_tablet_manager.cc:1397] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:36.791603 7209 raft_consensus.cc:348] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.791882 7209 raft_consensus.cc:374] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.791946 7209 raft_consensus.cc:729] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.792405 7209 consensus_queue.cc:260] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.792423 7210 raft_consensus.cc:348] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.792693 7210 raft_consensus.cc:374] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.792752 7210 raft_consensus.cc:729] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707d6246e844828a8a581fc15228167, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.793190 7210 consensus_queue.cc:260] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.793435 7209 ts_tablet_manager.cc:1428] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:49:36.793495 7211 raft_consensus.cc:348] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.793735 7211 raft_consensus.cc:374] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.793788 7211 raft_consensus.cc:729] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.793829 6906 heartbeater.cc:503] Master 127.4.219.190:46725 was elected leader, sending a full tablet report...
I20260925 22:49:36.794062 7046 heartbeater.cc:503] Master 127.4.219.190:46725 was elected leader, sending a full tablet report...
I20260925 22:49:36.794188 7210 ts_tablet_manager.cc:1428] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:36.794224 7211 consensus_queue.cc:260] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.795027 7186 heartbeater.cc:503] Master 127.4.219.190:46725 was elected leader, sending a full tablet report...
I20260925 22:49:36.795136 7211 ts_tablet_manager.cc:1428] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:49:36.919924 7216 raft_consensus.cc:482] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:36.920205 7216 raft_consensus.cc:504] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.921356 7216 leader_election.cc:288] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859), 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381)
I20260925 22:49:36.925215 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.921785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.4.219.130:50423)
0925 22:49:36.921957 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.921973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.922086 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.922355 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.922357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.922372 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:36.922505 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.922511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.923234 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.923239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.924051 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.924060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.924786 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.924790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.924908 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.924928 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.925017 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.925087 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:49:36.925215 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.921881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:50423 (local address 127.4.219.129:34381)
0925 22:49:36.921998 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.922005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.922020 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:36.922204 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.922207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.922255 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.922342 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:36.922620 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.923122 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.924192 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.924669 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.924793 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.924842 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.925032 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.925096 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:36.925860 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c133a5c90c3c431aa42ecb694d7f3c7f" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6" is_pre_election: true
I20260925 22:49:36.926209 6860 raft_consensus.cc:2454] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:36.926276 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.921785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33859 (local address 127.4.219.130:58743)
0925 22:49:36.922439 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.922452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.922575 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.922841 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.922843 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.922859 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:36.922980 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.922985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.923745 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.923748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.924989 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.924998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.925886 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.925890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.925989 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.926005 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.926098 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.926165 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":300,"thread_start_us":144,"threads_started":1}
I20260925 22:49:36.926337 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.921942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:58743 (local address 127.4.219.131:33859)
0925 22:49:36.922151 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.922158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.922536 (+ 378us) server_negotiation.cc:407] Connection header received
0925 22:49:36.922686 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.922689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.922738 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.922823 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:36.923085 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.923625 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.925107 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.925721 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.925885 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.925937 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.926135 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.926214 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 22:49:36.926712 6931 leader_election.cc:302] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:36.926990 7216 raft_consensus.cc:2781] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:36.926975 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c133a5c90c3c431aa42ecb694d7f3c7f" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9" is_pre_election: true
I20260925 22:49:36.927080 7216 raft_consensus.cc:482] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:36.927125 7216 raft_consensus.cc:3037] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:36.927260 7140 raft_consensus.cc:2454] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:36.928194 7216 raft_consensus.cc:504] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.928479 7216 leader_election.cc:288] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Requested vote from peers 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859), 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381)
I20260925 22:49:36.928833 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c133a5c90c3c431aa42ecb694d7f3c7f" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9"
I20260925 22:49:36.928844 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c133a5c90c3c431aa42ecb694d7f3c7f" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6"
I20260925 22:49:36.928960 6860 raft_consensus.cc:3037] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:36.928960 7140 raft_consensus.cc:3037] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:36.930222 6860 raft_consensus.cc:2454] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:36.930222 7140 raft_consensus.cc:2454] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:36.930506 6931 leader_election.cc:302] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:36.930721 7216 raft_consensus.cc:2781] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:36.930954 7216 raft_consensus.cc:686] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Becoming Leader. State: Replica: c707d6246e844828a8a581fc15228167, State: Running, Role: LEADER
I20260925 22:49:36.931272 7216 consensus_queue.cc:237] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } }
I20260925 22:49:36.934433 6703 catalog_manager.cc:5495] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 reported cstate change: term changed from 0 to 1, leader changed from <none> to c707d6246e844828a8a581fc15228167 (127.4.219.130). New cstate: current_term: 1 leader_uuid: "c707d6246e844828a8a581fc15228167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } health_report { overall_health: HEALTHY } } }
W20260925 22:49:36.959362 7187 tablet.cc:2369] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:49:36 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376575, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:36.983057 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.974115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.4.219.190:46725)
0925 22:49:36.974242 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:36.974248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:36.974404 (+ 156us) server_negotiation.cc:407] Connection header received
0925 22:49:36.974563 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:36.974566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:36.974610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:36.974694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:36.975348 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.975862 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.976714 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.976936 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.977071 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.979608 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:36.979611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:36.979613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:36.979636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:36.981284 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.981691 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.981694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.981695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.981782 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:36.982025 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:36.982027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:36.982028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:36.982261 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:36.982393 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:36.982711 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:49:36.982877 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:36.983103 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:36.974065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46725 (local address 127.0.0.1:37854)
0925 22:49:36.974321 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:36.974333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:36.974461 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:36.974710 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:36.974713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:36.975133 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:36.975230 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.975236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.975974 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.975979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.976587 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:36.976600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:36.977090 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:36.977097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:36.977236 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:36.978005 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:36.978027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:36.979471 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:36.981375 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.981385 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.981390 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.981607 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.981857 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:36.981859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:36.981860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:36.981949 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:36.982405 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:36.982414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:36.982537 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:49:36.982667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:49:36.982964 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":135,"thread_start_us":86,"threads_started":1}
W20260925 22:49:36.987453 7047 tablet.cc:2369] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:36.987486 6703 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:37854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:49:36.988229 6703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:36.993065 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 2780963ab31b4e079dafbe1b4cd80952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=788774a4bb394e57bea3b4aed2128b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:36.993185 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 2780963ab31b4e079dafbe1b4cd80952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=788774a4bb394e57bea3b4aed2128b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:36.993134 6980 tablet_service.cc:1467] Processing CreateTablet for tablet 2780963ab31b4e079dafbe1b4cd80952 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=788774a4bb394e57bea3b4aed2128b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:49:36.993801 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2780963ab31b4e079dafbe1b4cd80952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.993801 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2780963ab31b4e079dafbe1b4cd80952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.993811 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2780963ab31b4e079dafbe1b4cd80952. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:36.996230 7209 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: Bootstrap starting.
I20260925 22:49:36.996758 7210 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: Bootstrap starting.
I20260925 22:49:36.996904 7211 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: Bootstrap starting.
I20260925 22:49:36.997336 7209 tablet_bootstrap.cc:654] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.997655 7211 tablet_bootstrap.cc:654] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.997763 7210 tablet_bootstrap.cc:654] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:36.998899 7211 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: No bootstrap required, opened a new log
I20260925 22:49:36.998973 7209 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: No bootstrap required, opened a new log
I20260925 22:49:36.998973 7210 tablet_bootstrap.cc:492] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: No bootstrap required, opened a new log
I20260925 22:49:36.999013 7211 ts_tablet_manager.cc:1397] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:36.999060 7209 ts_tablet_manager.cc:1397] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:36.999082 7210 ts_tablet_manager.cc:1397] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:36.999414 7211 raft_consensus.cc:348] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:36.999543 7211 raft_consensus.cc:374] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.999509 7209 raft_consensus.cc:348] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:36.999507 7210 raft_consensus.cc:348] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:36.999584 7211 raft_consensus.cc:729] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.999611 7209 raft_consensus.cc:374] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.999622 7210 raft_consensus.cc:374] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:36.999660 7210 raft_consensus.cc:729] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707d6246e844828a8a581fc15228167, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.999660 7209 raft_consensus.cc:729] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Initialized, Role: FOLLOWER
I20260925 22:49:36.999724 7211 consensus_queue.cc:260] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:36.999790 7210 consensus_queue.cc:260] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:36.999816 7209 consensus_queue.cc:260] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.000030 7211 ts_tablet_manager.cc:1428] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:37.000092 7209 ts_tablet_manager.cc:1428] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:37.000099 7217 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.000094 7215 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.000238 7217 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.000272 7215 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.000859 7210 ts_tablet_manager.cc:1428] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:37.001390 7216 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.001516 7216 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 22:49:37.015427 6907 tablet.cc:2369] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:37.125629 7215 raft_consensus.cc:482] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:37.125765 7215 raft_consensus.cc:504] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.126936 7215 leader_election.cc:288] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c707d6246e844828a8a581fc15228167 (127.4.219.130:40507), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:37.131367 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.127347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:37673 (local address 127.4.219.130:40507)
0925 22:49:37.127506 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.127511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.127754 (+ 243us) server_negotiation.cc:407] Connection header received
0925 22:49:37.127864 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.127867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.127917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.128006 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:37.128309 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.129068 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.130272 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.130786 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.130875 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.130915 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.131171 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.131235 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:37.131419 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.127273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40507 (local address 127.4.219.129:37673)
0925 22:49:37.127656 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.127669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.127771 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.128034 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.128037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.128055 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:37.128193 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.128199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.129176 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.129180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.130144 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.130153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.130916 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.130920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.131018 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.131046 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.131130 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.131205 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":127,"threads_started":1}
I20260925 22:49:37.131650 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.127273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33859 (local address 127.4.219.129:53513)
0925 22:49:37.127784 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.127798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.127905 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.128186 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.128188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.128202 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:37.128736 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.128746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.129622 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.129624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.130491 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.130501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.131198 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.131202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.131296 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.131317 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.131396 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.131472 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":35,"thread_start_us":134,"threads_started":1}
I20260925 22:49:37.131688 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.127444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:53513 (local address 127.4.219.131:33859)
0925 22:49:37.127565 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.127571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.127896 (+ 325us) server_negotiation.cc:407] Connection header received
0925 22:49:37.128034 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.128037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.128084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.128164 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:37.128868 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.129508 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.130614 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.131089 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.131195 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.131238 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.131475 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.131560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:49:37.132097 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2780963ab31b4e079dafbe1b4cd80952" candidate_uuid: "14dbc5a27ade4518a15aec870df212c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707d6246e844828a8a581fc15228167" is_pre_election: true
I20260925 22:49:37.132179 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2780963ab31b4e079dafbe1b4cd80952" candidate_uuid: "14dbc5a27ade4518a15aec870df212c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9" is_pre_election: true
I20260925 22:49:37.132289 7140 raft_consensus.cc:2454] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14dbc5a27ade4518a15aec870df212c6 in term 0.
I20260925 22:49:37.132299 7000 raft_consensus.cc:2454] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14dbc5a27ade4518a15aec870df212c6 in term 0.
I20260925 22:49:37.132699 6792 leader_election.cc:302] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, 6ec1eb6139e34cb086a0982b776a60e9; no voters:
I20260925 22:49:37.132968 7215 raft_consensus.cc:2781] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:37.133047 7215 raft_consensus.cc:482] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:37.133100 7215 raft_consensus.cc:3037] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.134071 7215 raft_consensus.cc:504] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.134459 7215 leader_election.cc:288] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [CANDIDATE]: Term 1 election: Requested vote from peers c707d6246e844828a8a581fc15228167 (127.4.219.130:40507), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:37.134753 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2780963ab31b4e079dafbe1b4cd80952" candidate_uuid: "14dbc5a27ade4518a15aec870df212c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707d6246e844828a8a581fc15228167"
I20260925 22:49:37.134829 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2780963ab31b4e079dafbe1b4cd80952" candidate_uuid: "14dbc5a27ade4518a15aec870df212c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9"
I20260925 22:49:37.134902 7000 raft_consensus.cc:3037] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.134982 7140 raft_consensus.cc:3037] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.135824 7000 raft_consensus.cc:2454] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14dbc5a27ade4518a15aec870df212c6 in term 1.
I20260925 22:49:37.135878 7140 raft_consensus.cc:2454] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14dbc5a27ade4518a15aec870df212c6 in term 1.
I20260925 22:49:37.136147 6793 leader_election.cc:302] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:37.136334 7215 raft_consensus.cc:2781] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:37.136611 7215 raft_consensus.cc:686] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [term 1 LEADER]: Becoming Leader. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Running, Role: LEADER
I20260925 22:49:37.136957 7215 consensus_queue.cc:237] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.139258 7232 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: TxnStatusTablet state changed. Reason: New leader 14dbc5a27ade4518a15aec870df212c6. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.139389 7232 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:37.139930 7234 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:49:37.140153 7234 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:49:37.140702 6703 catalog_manager.cc:5495] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:37.157109 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.153396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.0.0.1:57474)
0925 22:49:37.153565 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.153581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.153686 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.154058 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.154061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.154068 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:37.154166 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.154171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.154893 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.154897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.155501 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.155508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.155946 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.155950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.156035 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.156085 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.156822 (+ 737us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.156837 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.156906 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.156999 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 22:49:37.157104 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.153489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.4.219.129:34381)
0925 22:49:37.153601 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.153608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.153622 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:37.153830 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.153835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.153924 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.154048 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.154279 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.154786 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.155618 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.155841 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.155956 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.156222 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.156727 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.156847 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.156914 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.156969 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:49:37.162299 7140 raft_consensus.cc:1261] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Refusing update from remote peer 14dbc5a27ade4518a15aec870df212c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.162348 7000 raft_consensus.cc:1261] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Refusing update from remote peer 14dbc5a27ade4518a15aec870df212c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.162953 7215 consensus_queue.cc:1035] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.163143 7232 consensus_queue.cc:1035] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.166118 7216 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: TxnStatusTablet state changed. Reason: New leader 14dbc5a27ade4518a15aec870df212c6. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.166224 7216 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.167284 7217 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: TxnStatusTablet state changed. Reason: New leader 14dbc5a27ade4518a15aec870df212c6. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.167389 7217 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.169728 7215 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.169837 7215 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:37.169950 7237 mvcc.cc:205] Tried to move back new op lower bound from 7333382460050247680 to 7333382459955605504. Current Snapshot: MvccSnapshot[applied={T|T < 7333382460050247680}]
I20260925 22:49:37.170289 7216 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.170389 7216 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P c707d6246e844828a8a581fc15228167: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.171518 7238 mvcc.cc:205] Tried to move back new op lower bound from 7333382460050247680 to 7333382459955605504. Current Snapshot: MvccSnapshot[applied={T|T < 7333382460050247680}]
I20260925 22:49:37.172235 7215 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.172334 7215 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 14dbc5a27ade4518a15aec870df212c6: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:49:37.173352 7239 mvcc.cc:205] Tried to move back new op lower bound from 7333382460050247680 to 7333382459955605504. Current Snapshot: MvccSnapshot[applied={T|T < 7333382460050247680}]
I20260925 22:49:37.173508 7217 tablet_replica.cc:439] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14dbc5a27ade4518a15aec870df212c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } } }
I20260925 22:49:37.173596 7217 tablet_replica.cc:442] T 2780963ab31b4e079dafbe1b4cd80952 P 6ec1eb6139e34cb086a0982b776a60e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:49:37.182893 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.178830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.4.219.130:40507)
0925 22:49:37.179147 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.179150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.179160 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:49:37.179215 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.179217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.179262 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.179339 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.179639 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.180135 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.181474 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.181679 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.181780 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.182084 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.182514 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.182600 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.182703 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.182745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260925 22:49:37.193744 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.190217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:50773 (local address 127.4.219.129:34381)
0925 22:49:37.190335 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.190338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.190636 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:49:37.190787 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.190791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.190840 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.190914 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.191165 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.191811 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.192684 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.192899 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.193019 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.193333 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.193399 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.193477 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.193561 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.193609 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:49:37.193759 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.190124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.4.219.130:50773)
0925 22:49:37.190549 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.190570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.190678 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.190932 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.190935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.190942 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:37.191065 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.191070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.191912 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.191919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.192587 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.192600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.193028 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.193032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.193134 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.193173 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.193480 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.193488 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.193554 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.193624 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260925 22:49:37.200302 7140 raft_consensus.cc:1261] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.200495 6860 raft_consensus.cc:1261] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.200795 7216 consensus_queue.cc:1035] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.201025 7219 consensus_queue.cc:1035] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.225281 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.221567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52017 (local address 127.4.219.130:40507)
0925 22:49:37.221707 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.221712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.221925 (+ 213us) server_negotiation.cc:407] Connection header received
0925 22:49:37.222086 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.222089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.222143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.222223 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.222602 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.223267 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.224201 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.224427 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.224565 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.224846 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.224923 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.225001 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.225092 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.225141 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:49:37.225298 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.221523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40507 (local address 127.4.219.129:52017)
0925 22:49:37.221844 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.221856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.221980 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.222344 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.222348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.222357 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:37.222483 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.222490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.223395 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.223401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.224077 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.224086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.224561 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.224565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.224660 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.224715 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.225016 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.225022 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.225082 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.225159 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":120,"threads_started":1}
I20260925 22:49:37.250722 6703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:37.250928 6703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:37.253736 6703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Sending DeleteTablet for 3 replicas of tablet c133a5c90c3c431aa42ecb694d7f3c7f
I20260925 22:49:37.254441 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:37 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40348
I20260925 22:49:37.254582 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:37 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:43560
I20260925 22:49:37.255282 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet c133a5c90c3c431aa42ecb694d7f3c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:37 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57458
I20260925 22:49:37.255748 7259 tablet_replica.cc:330] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: stopping tablet replica
I20260925 22:49:37.255872 6703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:49:37.256074 7259 raft_consensus.cc:2229] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:37.256253 7259 raft_consensus.cc:2258] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:49:37.256289 6703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:37.257644 7259 ts_tablet_manager.cc:1905] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:37.259440 7257 tablet_replica.cc:330] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: stopping tablet replica
I20260925 22:49:37.259753 7257 raft_consensus.cc:2229] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:37.259965 7257 raft_consensus.cc:2258] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:37.260883 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa8b40851a45bab17671406e0a42cc (DEFAULT_TABLE table=test-table [id=082d2a58fa9e44a1a9010133bfb3f174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:37.261181 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfa8b40851a45bab17671406e0a42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:37.261214 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa8b40851a45bab17671406e0a42cc (DEFAULT_TABLE table=test-table [id=082d2a58fa9e44a1a9010133bfb3f174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:37.261489 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfa8b40851a45bab17671406e0a42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:37.264144 7209 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Bootstrap starting.
I20260925 22:49:37.264654 7211 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Bootstrap starting.
I20260925 22:49:37.264784 6980 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfa8b40851a45bab17671406e0a42cc (DEFAULT_TABLE table=test-table [id=082d2a58fa9e44a1a9010133bfb3f174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:37.265059 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfa8b40851a45bab17671406e0a42cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:37.265604 7211 tablet_bootstrap.cc:654] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:37.267405 7210 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Bootstrap starting.
I20260925 22:49:37.267787 7209 tablet_bootstrap.cc:654] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:37.268330 7210 tablet_bootstrap.cc:654] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:37.269984 7257 ts_tablet_manager.cc:1905] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:37.274204 7257 ts_tablet_manager.cc:1918] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:37.274296 7257 log.cc:1196] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/c133a5c90c3c431aa42ecb694d7f3c7f
I20260925 22:49:37.274622 7257 ts_tablet_manager.cc:1939] T c133a5c90c3c431aa42ecb694d7f3c7f P 6ec1eb6139e34cb086a0982b776a60e9: Deleting consensus metadata
I20260925 22:49:37.275851 7211 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: No bootstrap required, opened a new log
I20260925 22:49:37.275949 7211 ts_tablet_manager.cc:1397] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260925 22:49:37.276407 7211 raft_consensus.cc:348] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.276571 7211 raft_consensus.cc:374] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:37.276628 7211 raft_consensus.cc:729] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Initialized, Role: FOLLOWER
I20260925 22:49:37.276762 7211 consensus_queue.cc:260] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.277042 7211 ts_tablet_manager.cc:1428] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:37.280620 7259 ts_tablet_manager.cc:1918] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:37.280654 7210 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: No bootstrap required, opened a new log
I20260925 22:49:37.280747 7259 log.cc:1196] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/c133a5c90c3c431aa42ecb694d7f3c7f
I20260925 22:49:37.280752 7210 ts_tablet_manager.cc:1397] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260925 22:49:37.281119 7259 ts_tablet_manager.cc:1939] T c133a5c90c3c431aa42ecb694d7f3c7f P 14dbc5a27ade4518a15aec870df212c6: Deleting consensus metadata
I20260925 22:49:37.281239 7210 raft_consensus.cc:348] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.281369 7210 raft_consensus.cc:374] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:37.281409 7210 raft_consensus.cc:729] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707d6246e844828a8a581fc15228167, State: Initialized, Role: FOLLOWER
I20260925 22:49:37.281553 7210 consensus_queue.cc:260] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.282461 6687 catalog_manager.cc:4841] TS 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381): tablet c133a5c90c3c431aa42ecb694d7f3c7f (table test-table [id=c6d67e3260c14abc995595637f808198]) successfully deleted
I20260925 22:49:37.284829 7210 ts_tablet_manager.cc:1428] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 22:49:37.286536 7258 tablet_replica.cc:330] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: stopping tablet replica
I20260925 22:49:37.286868 7258 raft_consensus.cc:2229] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:37.287240 7258 raft_consensus.cc:2258] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:37.288753 6688 catalog_manager.cc:4841] TS 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859): tablet c133a5c90c3c431aa42ecb694d7f3c7f (table test-table [id=c6d67e3260c14abc995595637f808198]) successfully deleted
I20260925 22:49:37.289288 7209 tablet_bootstrap.cc:492] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: No bootstrap required, opened a new log
I20260925 22:49:37.289390 7209 ts_tablet_manager.cc:1397] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20260925 22:49:37.289831 7209 raft_consensus.cc:348] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.289960 7209 raft_consensus.cc:374] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:37.290000 7209 raft_consensus.cc:729] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Initialized, Role: FOLLOWER
I20260925 22:49:37.290135 7209 consensus_queue.cc:260] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.290398 7209 ts_tablet_manager.cc:1428] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:37.291029 7258 ts_tablet_manager.cc:1905] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:37.294319 7258 ts_tablet_manager.cc:1918] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:37.294430 7258 log.cc:1196] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/c133a5c90c3c431aa42ecb694d7f3c7f
I20260925 22:49:37.294780 7258 ts_tablet_manager.cc:1939] T c133a5c90c3c431aa42ecb694d7f3c7f P c707d6246e844828a8a581fc15228167: Deleting consensus metadata
I20260925 22:49:37.296074 6689 catalog_manager.cc:4841] TS c707d6246e844828a8a581fc15228167 (127.4.219.130:40507): tablet c133a5c90c3c431aa42ecb694d7f3c7f (table test-table [id=c6d67e3260c14abc995595637f808198]) successfully deleted
I20260925 22:49:37.533702 7217 raft_consensus.cc:482] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:37.533850 7217 raft_consensus.cc:504] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.535094 7217 leader_election.cc:288] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), c707d6246e844828a8a581fc15228167 (127.4.219.130:40507)
I20260925 22:49:37.539341 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.535475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45211 (local address 127.4.219.129:34381)
0925 22:49:37.535607 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.535612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.535812 (+ 200us) server_negotiation.cc:407] Connection header received
0925 22:49:37.536016 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.536020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.536066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.536153 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:37.536447 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.537007 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.538206 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.538786 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.538922 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.538980 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.539148 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.539211 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:37.539361 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.535392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.4.219.131:45211)
0925 22:49:37.535693 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.535712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.535889 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.536164 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.536166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.536181 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:37.536328 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.536334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.537135 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.537141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.538089 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.538098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.538913 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.538917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.539012 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.539035 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.539130 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.539197 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":26,"spinlock_wait_cycles":24704,"thread_start_us":75,"threads_started":1}
I20260925 22:49:37.539450 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.535520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:51249 (local address 127.4.219.130:40507)
0925 22:49:37.535639 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.535643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.535829 (+ 186us) server_negotiation.cc:407] Connection header received
0925 22:49:37.536023 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.536026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.536082 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.536166 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:37.536433 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.536975 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.538203 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.538897 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.539028 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.539078 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.539250 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.539314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:49:37.539749 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.535415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40507 (local address 127.4.219.131:51249)
0925 22:49:37.535726 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.535737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.535895 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.536177 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.536179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.536195 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:37.536329 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.536334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.537082 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.537087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.538064 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.538073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.539028 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.539032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.539116 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.539140 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.539222 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.539291 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":67,"spinlock_wait_cycles":1792,"thread_start_us":74,"threads_started":1}
I20260925 22:49:37.540056 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfa8b40851a45bab17671406e0a42cc" candidate_uuid: "6ec1eb6139e34cb086a0982b776a60e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6" is_pre_election: true
I20260925 22:49:37.540247 6860 raft_consensus.cc:2454] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec1eb6139e34cb086a0982b776a60e9 in term 0.
I20260925 22:49:37.540249 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfa8b40851a45bab17671406e0a42cc" candidate_uuid: "6ec1eb6139e34cb086a0982b776a60e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707d6246e844828a8a581fc15228167" is_pre_election: true
I20260925 22:49:37.540416 7000 raft_consensus.cc:2454] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ec1eb6139e34cb086a0982b776a60e9 in term 0.
I20260925 22:49:37.540675 7071 leader_election.cc:302] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, 6ec1eb6139e34cb086a0982b776a60e9; no voters:
I20260925 22:49:37.540943 7217 raft_consensus.cc:2781] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:37.541002 7217 raft_consensus.cc:482] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:37.541038 7217 raft_consensus.cc:3037] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.542011 7217 raft_consensus.cc:504] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.542349 7217 leader_election.cc:288] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [CANDIDATE]: Term 1 election: Requested vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), c707d6246e844828a8a581fc15228167 (127.4.219.130:40507)
I20260925 22:49:37.542657 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfa8b40851a45bab17671406e0a42cc" candidate_uuid: "6ec1eb6139e34cb086a0982b776a60e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6"
I20260925 22:49:37.542796 6860 raft_consensus.cc:3037] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.542773 7000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfa8b40851a45bab17671406e0a42cc" candidate_uuid: "6ec1eb6139e34cb086a0982b776a60e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c707d6246e844828a8a581fc15228167"
I20260925 22:49:37.542917 7000 raft_consensus.cc:3037] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:37.543736 6860 raft_consensus.cc:2454] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec1eb6139e34cb086a0982b776a60e9 in term 1.
I20260925 22:49:37.543900 7000 raft_consensus.cc:2454] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec1eb6139e34cb086a0982b776a60e9 in term 1.
I20260925 22:49:37.544035 7071 leader_election.cc:302] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, 6ec1eb6139e34cb086a0982b776a60e9; no voters:
I20260925 22:49:37.544199 7217 raft_consensus.cc:2781] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:37.544363 7217 raft_consensus.cc:686] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Running, Role: LEADER
I20260925 22:49:37.544636 7217 consensus_queue.cc:237] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:37.546922 6703 catalog_manager.cc:5495] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "6ec1eb6139e34cb086a0982b776a60e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:37.570258 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.566786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.4.219.131:33859)
0925 22:49:37.566928 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.566932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.566944 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:37.567063 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.567065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.567112 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.567197 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.567428 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.567977 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.568837 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.569068 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.569171 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.569454 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.569885 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.570025 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.570070 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.570119 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:37.580333 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.577107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:43505 (local address 127.4.219.129:34381)
0925 22:49:37.577248 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.577251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.577357 (+ 106us) server_negotiation.cc:407] Connection header received
0925 22:49:37.577501 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.577504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.577560 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.577632 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.577879 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.578522 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.579362 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.579616 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.579699 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.579953 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.580019 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.580084 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.580186 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.580228 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:37.580781 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.576995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.4.219.131:43505)
0925 22:49:37.577281 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.577294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.577376 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.577647 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.577649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.577655 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:37.577752 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.577756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.578650 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.578652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.579257 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.579264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.579715 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.579717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.579787 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.579824 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.580100 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.580104 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.580152 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.580212 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":110,"threads_started":1}
I20260925 22:49:37.586627 6860 raft_consensus.cc:1261] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Refusing update from remote peer 6ec1eb6139e34cb086a0982b776a60e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.586635 7000 raft_consensus.cc:1261] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Refusing update from remote peer 6ec1eb6139e34cb086a0982b776a60e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:37.587188 7217 consensus_queue.cc:1035] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.587361 7265 consensus_queue.cc:1035] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:37.604501 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.601317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:40819 (local address 127.4.219.131:33859)
0925 22:49:37.601421 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:37.601425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:37.601449 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:49:37.601604 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:37.601607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:37.601655 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:37.601713 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:37.601908 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.602391 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.603395 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.603678 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.603794 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.604033 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.604113 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.604186 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:37.604302 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 22:49:37.604348 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:49:37.604477 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:37.601233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33859 (local address 127.4.219.129:40819)
0925 22:49:37.601366 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:37.601380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:37.601499 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:37.601747 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:37.601750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:37.601757 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:49:37.601835 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.601840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.602484 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.602486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.603260 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:37.603269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:37.603802 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:37.603805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:37.603865 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:37.603890 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:49:37.604210 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:49:37.604217 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:49:37.604286 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:49:37.604363 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:49:37.605250 7239 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:49:39.160742 6702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:39.160899 6702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:39.163529 6702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Sending DeleteTablet for 3 replicas of tablet 5bfa8b40851a45bab17671406e0a42cc
I20260925 22:49:39.164222 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa8b40851a45bab17671406e0a42cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57458
I20260925 22:49:39.164237 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa8b40851a45bab17671406e0a42cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:43560
I20260925 22:49:39.164438 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfa8b40851a45bab17671406e0a42cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40348
I20260925 22:49:39.164808 7300 tablet_replica.cc:330] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: stopping tablet replica
I20260925 22:49:39.164939 7300 raft_consensus.cc:2229] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:39.165259 7300 raft_consensus.cc:2258] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.165584 7298 tablet_replica.cc:330] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: stopping tablet replica
I20260925 22:49:39.165750 7298 raft_consensus.cc:2229] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:39.165885 7298 raft_consensus.cc:2258] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.165995 7300 ts_tablet_manager.cc:1905] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:39.164757 7299 tablet_replica.cc:330] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: stopping tablet replica
I20260925 22:49:39.166157 7299 raft_consensus.cc:2229] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:39.166290 7299 raft_consensus.cc:2258] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.166865 7298 ts_tablet_manager.cc:1905] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:39.168416 7300 ts_tablet_manager.cc:1918] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:49:39.168507 7300 log.cc:1196] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/5bfa8b40851a45bab17671406e0a42cc
I20260925 22:49:39.168854 7300 ts_tablet_manager.cc:1939] T 5bfa8b40851a45bab17671406e0a42cc P 6ec1eb6139e34cb086a0982b776a60e9: Deleting consensus metadata
I20260925 22:49:39.169435 7299 ts_tablet_manager.cc:1905] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:39.169719 7298 ts_tablet_manager.cc:1918] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:49:39.169723 6688 catalog_manager.cc:4841] TS 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859): tablet 5bfa8b40851a45bab17671406e0a42cc (table test-table [id=082d2a58fa9e44a1a9010133bfb3f174]) successfully deleted
I20260925 22:49:39.169800 7298 log.cc:1196] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/5bfa8b40851a45bab17671406e0a42cc
I20260925 22:49:39.170077 7298 ts_tablet_manager.cc:1939] T 5bfa8b40851a45bab17671406e0a42cc P c707d6246e844828a8a581fc15228167: Deleting consensus metadata
I20260925 22:49:39.170817 6689 catalog_manager.cc:4841] TS c707d6246e844828a8a581fc15228167 (127.4.219.130:40507): tablet 5bfa8b40851a45bab17671406e0a42cc (table test-table [id=082d2a58fa9e44a1a9010133bfb3f174]) successfully deleted
I20260925 22:49:39.171998 7299 ts_tablet_manager.cc:1918] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:49:39.172077 7299 log.cc:1196] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/5bfa8b40851a45bab17671406e0a42cc
I20260925 22:49:39.172335 7299 ts_tablet_manager.cc:1939] T 5bfa8b40851a45bab17671406e0a42cc P 14dbc5a27ade4518a15aec870df212c6: Deleting consensus metadata
I20260925 22:49:39.173002 6687 catalog_manager.cc:4841] TS 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381): tablet 5bfa8b40851a45bab17671406e0a42cc (table test-table [id=082d2a58fa9e44a1a9010133bfb3f174]) successfully deleted
Sep 25 22:49:39 dist-test-slave-kb1g krb5kdc[6658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:49:39 dist-test-slave-kb1g krb5kdc[6658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:39.200309 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:39.185752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.4.219.190:46725)
0925 22:49:39.186020 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:39.186026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:39.186041 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:39.186149 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:39.186152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:39.186197 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:39.186284 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:39.187039 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:39.187656 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:39.188505 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:39.189014 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:39.189105 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:39.196843 (+ 7738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:39.196846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:39.196849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:39.196881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:39.198511 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:39.198991 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:39.198994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:39.198995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:39.199097 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:39.199379 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:39.199381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:39.199382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:39.199581 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:39.199692 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:39.200033 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:49:39.200152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260925 22:49:39.203173 6702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:39.203665 6702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:39.208163 6838 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (DEFAULT_TABLE table=test-table [id=078cd39112fe4f75b875041c66142222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.208325 7120 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (DEFAULT_TABLE table=test-table [id=078cd39112fe4f75b875041c66142222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.208374 6980 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (DEFAULT_TABLE table=test-table [id=078cd39112fe4f75b875041c66142222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.208467 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.208634 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.208674 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.210695 7309 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Bootstrap starting.
I20260925 22:49:39.211086 7311 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Bootstrap starting.
I20260925 22:49:39.211501 7310 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Bootstrap starting.
I20260925 22:49:39.211822 7309 tablet_bootstrap.cc:654] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.212006 7311 tablet_bootstrap.cc:654] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.212469 7310 tablet_bootstrap.cc:654] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.213073 7309 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: No bootstrap required, opened a new log
I20260925 22:49:39.213164 7309 ts_tablet_manager.cc:1397] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:39.213393 7311 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: No bootstrap required, opened a new log
I20260925 22:49:39.213497 7311 ts_tablet_manager.cc:1397] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:39.213656 7309 raft_consensus.cc:348] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.213784 7309 raft_consensus.cc:374] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.213819 7309 raft_consensus.cc:729] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707d6246e844828a8a581fc15228167, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.213950 7309 consensus_queue.cc:260] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.213917 7311 raft_consensus.cc:348] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.214046 7311 raft_consensus.cc:374] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.214085 7311 raft_consensus.cc:729] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.214124 7310 tablet_bootstrap.cc:492] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: No bootstrap required, opened a new log
I20260925 22:49:39.214201 7310 ts_tablet_manager.cc:1397] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:39.214224 7311 consensus_queue.cc:260] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.214318 7309 ts_tablet_manager.cc:1428] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:39.214603 7310 raft_consensus.cc:348] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.214730 7310 raft_consensus.cc:374] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.214761 7310 raft_consensus.cc:729] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.214836 7311 ts_tablet_manager.cc:1428] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:39.214917 7310 consensus_queue.cc:260] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.215198 7310 ts_tablet_manager.cc:1428] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:39.306877 7315 raft_consensus.cc:482] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:39.307035 7315 raft_consensus.cc:504] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.307540 7315 leader_election.cc:288] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:39.307996 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2d9c91b0c44ebf9e4cec9c73bf5b8d" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9" is_pre_election: true
I20260925 22:49:39.308032 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2d9c91b0c44ebf9e4cec9c73bf5b8d" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6" is_pre_election: true
I20260925 22:49:39.308156 6860 raft_consensus.cc:2454] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:39.308154 7140 raft_consensus.cc:2454] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:39.308521 6931 leader_election.cc:302] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:39.308744 7315 raft_consensus.cc:2781] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:39.308811 7315 raft_consensus.cc:482] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:39.308856 7315 raft_consensus.cc:3037] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.309901 7315 raft_consensus.cc:504] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.310242 7315 leader_election.cc:288] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Requested vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:39.310562 7140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2d9c91b0c44ebf9e4cec9c73bf5b8d" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9"
I20260925 22:49:39.310578 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2d9c91b0c44ebf9e4cec9c73bf5b8d" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6"
I20260925 22:49:39.310691 7140 raft_consensus.cc:3037] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.310738 6860 raft_consensus.cc:3037] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.311585 7140 raft_consensus.cc:2454] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:39.311784 6860 raft_consensus.cc:2454] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:39.311863 6932 leader_election.cc:302] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ec1eb6139e34cb086a0982b776a60e9, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:39.312055 7315 raft_consensus.cc:2781] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:39.312268 7315 raft_consensus.cc:686] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Becoming Leader. State: Replica: c707d6246e844828a8a581fc15228167, State: Running, Role: LEADER
I20260925 22:49:39.312417 7315 consensus_queue.cc:237] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.313973 6701 catalog_manager.cc:5495] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 reported cstate change: term changed from 0 to 1, leader changed from <none> to c707d6246e844828a8a581fc15228167 (127.4.219.130). New cstate: current_term: 1 leader_uuid: "c707d6246e844828a8a581fc15228167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:39.345973 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:39.342299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.4.219.130:40507)
0925 22:49:39.342678 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:39.342691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:39.342707 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:49:39.342754 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:39.342757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:39.342811 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:39.342904 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:39.343183 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:39.343784 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:39.344804 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:39.345073 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:39.345191 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:39.345458 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:39.345542 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:39.345629 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:39.345754 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:49:39.345809 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":95,"threads_started":1}
I20260925 22:49:39.351930 6860 raft_consensus.cc:1261] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:39.352032 7140 raft_consensus.cc:1261] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:39.352348 7315 consensus_queue.cc:1035] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:39.352528 7316 consensus_queue.cc:1035] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:39.384743 6701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:39.384891 6701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:39.386713 6701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Sending DeleteTablet for 3 replicas of tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d
I20260925 22:49:39.387238 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:43560
I20260925 22:49:39.387339 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40348
I20260925 22:49:39.387454 7298 tablet_replica.cc:330] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: stopping tablet replica
I20260925 22:49:39.387449 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:39 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57458
I20260925 22:49:39.387503 7300 tablet_replica.cc:330] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: stopping tablet replica
I20260925 22:49:39.387583 7298 raft_consensus.cc:2229] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:39.387614 7300 raft_consensus.cc:2229] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:39.387706 7300 raft_consensus.cc:2258] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.387784 7298 raft_consensus.cc:2258] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.387872 7299 tablet_replica.cc:330] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: stopping tablet replica
I20260925 22:49:39.387995 7299 raft_consensus.cc:2229] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:39.388119 7299 raft_consensus.cc:2258] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:39.388581 7298 ts_tablet_manager.cc:1905] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:39.388700 6701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:49:39.388801 7299 ts_tablet_manager.cc:1905] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:39.388887 7300 ts_tablet_manager.cc:1905] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:49:39.389143 6701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:39.391234 7298 ts_tablet_manager.cc:1918] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:39.391301 7298 log.cc:1196] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/2b2d9c91b0c44ebf9e4cec9c73bf5b8d
I20260925 22:49:39.391443 7300 ts_tablet_manager.cc:1918] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:39.391515 7300 log.cc:1196] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/2b2d9c91b0c44ebf9e4cec9c73bf5b8d
I20260925 22:49:39.391582 7298 ts_tablet_manager.cc:1939] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P c707d6246e844828a8a581fc15228167: Deleting consensus metadata
I20260925 22:49:39.391731 7300 ts_tablet_manager.cc:1939] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 6ec1eb6139e34cb086a0982b776a60e9: Deleting consensus metadata
I20260925 22:49:39.392292 6688 catalog_manager.cc:4841] TS 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859): tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (table test-table [id=078cd39112fe4f75b875041c66142222]) successfully deleted
I20260925 22:49:39.393379 7299 ts_tablet_manager.cc:1918] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:49:39.393473 7299 log.cc:1196] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/2b2d9c91b0c44ebf9e4cec9c73bf5b8d
I20260925 22:49:39.393786 7299 ts_tablet_manager.cc:1939] T 2b2d9c91b0c44ebf9e4cec9c73bf5b8d P 14dbc5a27ade4518a15aec870df212c6: Deleting consensus metadata
I20260925 22:49:39.394143 6838 tablet_service.cc:1467] Processing CreateTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 (DEFAULT_TABLE table=test-table [id=eaef9fd8f6e040a9b33c5da63b8836ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.394430 6838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c497fc6c705b4360a8efec0b4cea2be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.394404 7120 tablet_service.cc:1467] Processing CreateTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 (DEFAULT_TABLE table=test-table [id=eaef9fd8f6e040a9b33c5da63b8836ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.394532 6687 catalog_manager.cc:4841] TS 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381): tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (table test-table [id=078cd39112fe4f75b875041c66142222]) successfully deleted
I20260925 22:49:39.394637 7120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c497fc6c705b4360a8efec0b4cea2be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.394650 6689 catalog_manager.cc:4841] TS c707d6246e844828a8a581fc15228167 (127.4.219.130:40507): tablet 2b2d9c91b0c44ebf9e4cec9c73bf5b8d (table test-table [id=078cd39112fe4f75b875041c66142222]) successfully deleted
I20260925 22:49:39.394961 6980 tablet_service.cc:1467] Processing CreateTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 (DEFAULT_TABLE table=test-table [id=eaef9fd8f6e040a9b33c5da63b8836ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:39.395219 6980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c497fc6c705b4360a8efec0b4cea2be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:39.396685 7310 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Bootstrap starting.
I20260925 22:49:39.396746 7311 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: Bootstrap starting.
I20260925 22:49:39.397204 7309 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Bootstrap starting.
I20260925 22:49:39.397591 7311 tablet_bootstrap.cc:654] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.397635 7310 tablet_bootstrap.cc:654] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.398103 7309 tablet_bootstrap.cc:654] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:39.398604 7311 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: No bootstrap required, opened a new log
I20260925 22:49:39.398692 7311 ts_tablet_manager.cc:1397] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:39.399062 7311 raft_consensus.cc:348] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.399178 7311 raft_consensus.cc:374] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.399221 7311 raft_consensus.cc:729] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14dbc5a27ade4518a15aec870df212c6, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.399345 7311 consensus_queue.cc:260] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.399662 7311 ts_tablet_manager.cc:1428] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:39.400672 7310 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: No bootstrap required, opened a new log
I20260925 22:49:39.400768 7310 ts_tablet_manager.cc:1397] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:39.400770 7309 tablet_bootstrap.cc:492] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: No bootstrap required, opened a new log
I20260925 22:49:39.400856 7309 ts_tablet_manager.cc:1397] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:39.401260 7309 raft_consensus.cc:348] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.401376 7309 raft_consensus.cc:374] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.401242 7310 raft_consensus.cc:348] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.401407 7309 raft_consensus.cc:729] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c707d6246e844828a8a581fc15228167, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.401449 7310 raft_consensus.cc:374] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:39.401520 7310 raft_consensus.cc:729] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec1eb6139e34cb086a0982b776a60e9, State: Initialized, Role: FOLLOWER
I20260925 22:49:39.401558 7309 consensus_queue.cc:260] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.401710 7310 consensus_queue.cc:260] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.401832 7309 ts_tablet_manager.cc:1428] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:39.402446 7310 ts_tablet_manager.cc:1428] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:39.470521 7323 raft_consensus.cc:482] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:39.470654 7323 raft_consensus.cc:504] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.471011 7323 leader_election.cc:288] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:39.471547 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c497fc6c705b4360a8efec0b4cea2be3" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6" is_pre_election: true
I20260925 22:49:39.471549 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c497fc6c705b4360a8efec0b4cea2be3" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9" is_pre_election: true
I20260925 22:49:39.471715 7139 raft_consensus.cc:2454] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:39.471715 6860 raft_consensus.cc:2454] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 0.
I20260925 22:49:39.472076 6932 leader_election.cc:302] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, 6ec1eb6139e34cb086a0982b776a60e9, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:39.472301 7323 raft_consensus.cc:2781] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:39.472379 7323 raft_consensus.cc:482] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:39.472432 7323 raft_consensus.cc:3037] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.473460 7323 raft_consensus.cc:504] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.473796 7323 leader_election.cc:288] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Requested vote from peers 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381), 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859)
I20260925 22:49:39.474215 6860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c497fc6c705b4360a8efec0b4cea2be3" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14dbc5a27ade4518a15aec870df212c6"
I20260925 22:49:39.474213 7139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c497fc6c705b4360a8efec0b4cea2be3" candidate_uuid: "c707d6246e844828a8a581fc15228167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec1eb6139e34cb086a0982b776a60e9"
I20260925 22:49:39.474362 6860 raft_consensus.cc:3037] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.474365 7139 raft_consensus.cc:3037] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:39.475464 6860 raft_consensus.cc:2454] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:39.475638 7139 raft_consensus.cc:2454] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c707d6246e844828a8a581fc15228167 in term 1.
I20260925 22:49:39.475757 6931 leader_election.cc:302] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14dbc5a27ade4518a15aec870df212c6, c707d6246e844828a8a581fc15228167; no voters:
I20260925 22:49:39.475999 7323 raft_consensus.cc:2781] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:39.476117 7323 raft_consensus.cc:686] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Becoming Leader. State: Replica: c707d6246e844828a8a581fc15228167, State: Running, Role: LEADER
I20260925 22:49:39.476260 7323 consensus_queue.cc:237] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } }
I20260925 22:49:39.478224 6701 catalog_manager.cc:5495] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 reported cstate change: term changed from 0 to 1, leader changed from <none> to c707d6246e844828a8a581fc15228167 (127.4.219.130). New cstate: current_term: 1 leader_uuid: "c707d6246e844828a8a581fc15228167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c707d6246e844828a8a581fc15228167" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 } health_report { overall_health: UNKNOWN } } }
I20260925 22:49:39.507124 7139 raft_consensus.cc:1261] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:39.507333 6860 raft_consensus.cc:1261] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6 [term 1 FOLLOWER]: Refusing update from remote peer c707d6246e844828a8a581fc15228167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:39.507545 7323 consensus_queue.cc:1035] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec1eb6139e34cb086a0982b776a60e9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:39.507696 7316 consensus_queue.cc:1035] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14dbc5a27ade4518a15aec870df212c6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:49:39.520004 7280 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:49:41.069617 6702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:41.069763 6702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:41.072158 6702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf9de2c9cb24a5fb638581f7a575864: Sending DeleteTablet for 3 replicas of tablet c497fc6c705b4360a8efec0b4cea2be3
I20260925 22:49:41.072801 6838 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:41 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57458
I20260925 22:49:41.072803 6980 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:41 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:43560
I20260925 22:49:41.072824 7120 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fc6c705b4360a8efec0b4cea2be3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:41 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40348
I20260925 22:49:41.073179 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6775
I20260925 22:49:41.073231 7351 tablet_replica.cc:330] T c497fc6c705b4360a8efec0b4cea2be3 P 14dbc5a27ade4518a15aec870df212c6: stopping tablet replica
I20260925 22:49:41.074362 7350 tablet_replica.cc:330] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: stopping tablet replica
I20260925 22:49:41.074504 7350 raft_consensus.cc:2229] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:41.074734 7350 raft_consensus.cc:2258] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:41.074985 7352 tablet_replica.cc:330] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: stopping tablet replica
I20260925 22:49:41.075124 7352 raft_consensus.cc:2229] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:41.075242 7352 raft_consensus.cc:2258] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:41.075354 7350 ts_tablet_manager.cc:1905] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:41.075839 7352 ts_tablet_manager.cc:1905] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:41.078128 7350 ts_tablet_manager.cc:1918] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:49:41.078208 7350 log.cc:1196] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/c497fc6c705b4360a8efec0b4cea2be3
I20260925 22:49:41.078527 7350 ts_tablet_manager.cc:1939] T c497fc6c705b4360a8efec0b4cea2be3 P c707d6246e844828a8a581fc15228167: Deleting consensus metadata
I20260925 22:49:41.078617 7352 ts_tablet_manager.cc:1918] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:49:41.078691 7352 log.cc:1196] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/c497fc6c705b4360a8efec0b4cea2be3
I20260925 22:49:41.078930 7352 ts_tablet_manager.cc:1939] T c497fc6c705b4360a8efec0b4cea2be3 P 6ec1eb6139e34cb086a0982b776a60e9: Deleting consensus metadata
I20260925 22:49:41.079340 6689 catalog_manager.cc:4841] TS c707d6246e844828a8a581fc15228167 (127.4.219.130:40507): tablet c497fc6c705b4360a8efec0b4cea2be3 (table test-table [id=eaef9fd8f6e040a9b33c5da63b8836ec]) successfully deleted
I20260925 22:49:41.079546 6688 catalog_manager.cc:4841] TS 6ec1eb6139e34cb086a0982b776a60e9 (127.4.219.131:33859): tablet c497fc6c705b4360a8efec0b4cea2be3 (table test-table [id=eaef9fd8f6e040a9b33c5da63b8836ec]) successfully deleted
W20260925 22:49:41.081506 6687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:34381 (error 108)
I20260925 22:49:41.082033 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6915
I20260925 22:49:41.083194 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:41.082274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34381 (local address 127.0.0.1:57490)
0925 22:49:41.083042 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:41.083109 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":105,"threads_started":1}
W20260925 22:49:41.083336 6687 catalog_manager.cc:4584] TS 14dbc5a27ade4518a15aec870df212c6 (127.4.219.129:34381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c497fc6c705b4360a8efec0b4cea2be3: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34381: connect: Connection refused (error 111)
I20260925 22:49:41.089991 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7055
I20260925 22:49:41.096614 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 6671
2026-09-25T22:49:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6457 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:41 dist-test-slave-kb1g krb5kdc[7357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:41 dist-test-slave-kb1g krb5kdc[7357](info): set up 2 sockets
Sep 25 22:49:41 dist-test-slave-kb1g krb5kdc[7357](info): commencing operation
krb5kdc: starting...
W20260925 22:49:42.144747 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:42.273144 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37295
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:37295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.384593 7370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.384925 7370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.384968 7370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.389237 7370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:42.389300 7370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.389320 7370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.389338 7370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:42.389353 7370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:42.394825 7370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:37295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:42.395979 7370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.397086 7370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.405849 7376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.405829 7370 server_base.cc:1034] running on GCE node
W20260925 22:49:42.405890 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.405804 7375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.406719 7370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.407872 7370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.409065 7370 hybrid_clock.cc:648] HybridClock initialized: now 1790376582409052 us; error 27 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.412039 7370 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:42.413311 7370 webserver.cc:460] Webserver started at http://127.4.219.190:34609/ using document root <none> and password file <none>
I20260925 22:49:42.414131 7370 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.414247 7370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.414559 7370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.417217 7370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "02715248cfd34e089221764e29d65109"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "c06ed2c92d9c024c32574a310077cadc"
server_key_iv: "78edfbb51a9421dba8878f08a0a81fca"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.417836 7370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "02715248cfd34e089221764e29d65109"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "c06ed2c92d9c024c32574a310077cadc"
server_key_iv: "78edfbb51a9421dba8878f08a0a81fca"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.423414 7370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:42.426383 7385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.427759 7370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:42.427938 7370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "02715248cfd34e089221764e29d65109"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "c06ed2c92d9c024c32574a310077cadc"
server_key_iv: "78edfbb51a9421dba8878f08a0a81fca"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.428063 7370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.451629 7370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.454679 7370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.454916 7370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.462182 7370 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:37295
I20260925 22:49:42.462240 7437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:37295 every 8 connection(s)
I20260925 22:49:42.463188 7370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:42.466033 7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:42.470069 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7370
I20260925 22:49:42.470220 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:42.470518 4974 external_mini_cluster.cc:1428] Setting key ea44f8e307b62866187d601b2a5de0f6
I20260925 22:49:42.472072 7438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109: Bootstrap starting.
I20260925 22:49:42.474663 7438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:42.475400 7438 log.cc:824] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:42.477272 7438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109: No bootstrap required, opened a new log
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:42.481117 7438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } }
I20260925 22:49:42.481376 7438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:42.481456 7438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02715248cfd34e089221764e29d65109, State: Initialized, Role: FOLLOWER
I20260925 22:49:42.481995 7438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } }
I20260925 22:49:42.482167 7438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:42.482240 7438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:42.482359 7438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:42.483515 7438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } }
I20260925 22:49:42.483882 7438 leader_election.cc:302] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02715248cfd34e089221764e29d65109; no voters:
I20260925 22:49:42.484226 7438 leader_election.cc:288] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:42.484318 7443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:42.484552 7443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [term 1 LEADER]: Becoming Leader. State: Replica: 02715248cfd34e089221764e29d65109, State: Running, Role: LEADER
I20260925 22:49:42.484800 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.471880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.4.219.190:37295)
0925 22:49:42.472293 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.472308 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.472343 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:49:42.473027 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.473046 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.473391 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.473675 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.474577 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.475662 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.476508 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.477025 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.477197 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.479699 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.479704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.479717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.479743 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.481437 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.481944 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.481949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.481953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.482062 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.482413 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.482415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.482417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.482840 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.482992 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.483377 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.483713 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":141,"threads_started":1}
I20260925 22:49:42.484871 7443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } }
I20260925 22:49:42.485638 7438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:42.486064 7443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02715248cfd34e089221764e29d65109. Latest consensus state: current_term: 1 leader_uuid: "02715248cfd34e089221764e29d65109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } } }
I20260925 22:49:42.486202 7443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:42.486910 7444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02715248cfd34e089221764e29d65109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02715248cfd34e089221764e29d65109" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37295 } } }
I20260925 22:49:42.487010 7444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:42.487406 7447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:42.490628 7447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:42.496796 7447 catalog_manager.cc:1347] Generated new cluster ID: ad72cf7bc59a41d0a18390bedd508c5a
I20260925 22:49:42.496868 7447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:42.516357 7447 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:42.517513 7447 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:42.528435 7447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02715248cfd34e089221764e29d65109: Generated new TSK 0
I20260925 22:49:42.529233 7447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:42.561022 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.678733 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.678993 7464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.679035 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.682895 7464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.682941 7464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.682996 7464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:42.688210 7464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:42.689172 7464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.690112 7464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.699342 7472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.699366 7469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.699510 7464 server_base.cc:1034] running on GCE node
W20260925 22:49:42.699414 7470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.700083 7464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.700799 7464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.701984 7464 hybrid_clock.cc:648] HybridClock initialized: now 1790376582701958 us; error 38 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.704737 7464 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:42.705508 7464 webserver.cc:460] Webserver started at http://127.4.219.129:44711/ using document root <none> and password file <none>
I20260925 22:49:42.706005 7464 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.706063 7464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.706223 7464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.708205 7464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "cf31a59e51484edd91f37a83225e1cb2"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "375b47f452941a26f2bd9581d848aba3"
server_key_iv: "7ef2e0f6d071174a4fc3e133dad4d62d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.708994 7464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "cf31a59e51484edd91f37a83225e1cb2"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "375b47f452941a26f2bd9581d848aba3"
server_key_iv: "7ef2e0f6d071174a4fc3e133dad4d62d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.712971 7464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:49:42.715602 7479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.717011 7464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:42.717154 7464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "cf31a59e51484edd91f37a83225e1cb2"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "375b47f452941a26f2bd9581d848aba3"
server_key_iv: "7ef2e0f6d071174a4fc3e133dad4d62d"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.717271 7464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.735525 7464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.739181 7464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.739425 7464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.740095 7464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:42.741138 7464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:42.741216 7464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.741298 7464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:42.741345 7464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.751046 7464 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:43493
I20260925 22:49:42.751117 7592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:43493 every 8 connection(s)
I20260925 22:49:42.752045 7464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:42.758108 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7464
I20260925 22:49:42.758229 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:42.758529 4974 external_mini_cluster.cc:1428] Setting key 1d716dde78be300cd897bfabf2628189
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:42.767504 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.754119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:49517 (local address 127.4.219.190:37295)
0925 22:49:42.754270 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.754274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.755130 (+ 856us) server_negotiation.cc:407] Connection header received
0925 22:49:42.756611 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.756615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.756674 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.756793 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.758142 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.758843 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.760400 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.760656 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.760746 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.763670 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.763674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.763677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.763711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.765324 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.765889 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.765893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.765895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.765978 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.766351 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.766355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.766357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.766561 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.766678 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.767195 (+ 517us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.767328 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:42.768474 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.754518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37295 (local address 127.4.219.129:49517)
0925 22:49:42.754972 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.755017 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.756351 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.756977 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.756984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.757361 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:42.757951 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.757974 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.758984 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.758987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.760251 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.760263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.760823 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.760830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.761106 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.761766 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:42.761794 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:42.763490 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:42.765464 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.765472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.765488 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.765770 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.766087 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.766091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.766093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.766244 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.766677 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:42.766686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:42.766949 (+ 263us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.767155 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.767516 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260925 22:49:42.769884 7593 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37295
I20260925 22:49:42.770103 7593 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:42.770733 7593 heartbeater.cc:511] Master 127.4.219.190:37295 requested a full tablet report, sending...
I20260925 22:49:42.772349 7402 ts_manager.cc:194] Registered new tserver with Master: cf31a59e51484edd91f37a83225e1cb2 (127.4.219.129:43493)
I20260925 22:49:42.773525 7402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:49517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:42.788918 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:42.899739 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:42.899979 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:42.900027 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:42.904131 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:42.904186 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:42.904243 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:42.909590 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:42.910629 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:42.911648 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:42.921698 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.921682 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:42.921685 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:42.921998 7598 server_base.cc:1034] running on GCE node
I20260925 22:49:42.922472 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:42.923077 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:42.924245 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790376582924192 us; error 76 us; skew 500 ppm
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:42.927255 7598 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:42.928102 7598 webserver.cc:460] Webserver started at http://127.4.219.130:43039/ using document root <none> and password file <none>
I20260925 22:49:42.928761 7598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:42.928848 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:42.929023 7598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:42.931077 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "a9a8e86cf2cf4bafad8b038ef1ac503a"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "00e11757f213c0feadd49d0d7b5255a8"
server_key_iv: "bef2f6ba1cca9d872540b9f827fcef42"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.931656 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "a9a8e86cf2cf4bafad8b038ef1ac503a"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "00e11757f213c0feadd49d0d7b5255a8"
server_key_iv: "bef2f6ba1cca9d872540b9f827fcef42"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.935667 7598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:42.938119 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.939353 7598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:42.939488 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "a9a8e86cf2cf4bafad8b038ef1ac503a"
format_stamp: "Formatted at 2026-09-25 22:49:42 on dist-test-slave-kb1g"
server_key: "00e11757f213c0feadd49d0d7b5255a8"
server_key_iv: "bef2f6ba1cca9d872540b9f827fcef42"
server_key_version: "encryptionkey@0"
I20260925 22:49:42.939595 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:42.952093 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:42.955020 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:42.955222 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:42.955878 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:42.956774 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:42.956835 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.956889 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:42.956946 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:42.965596 7598 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:43597
I20260925 22:49:42.965626 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:43597 every 8 connection(s)
I20260925 22:49:42.966495 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:42.975231 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7598
I20260925 22:49:42.975353 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:42.975692 4974 external_mini_cluster.cc:1428] Setting key 2acb3d7dd839ead487feb72751787f82
Sep 25 22:49:42 dist-test-slave-kb1g krb5kdc[7357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376582, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:42.980499 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.968260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:60579 (local address 127.4.219.190:37295)
0925 22:49:42.968418 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:42.968423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:42.969393 (+ 970us) server_negotiation.cc:407] Connection header received
0925 22:49:42.970664 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:42.970667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:42.970714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:42.970807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:42.971937 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.972477 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.973661 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.973908 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.973994 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.976841 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:42.976844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:42.976847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:42.976870 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:42.978463 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.978991 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.978996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.978998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.979080 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:42.979426 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:42.979431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:42.979433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:42.979617 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:42.979722 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:42.980224 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 22:49:42.980343 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:49:42.981413 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:42.968777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37295 (local address 127.4.219.130:60579)
0925 22:49:42.969234 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:42.969275 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:42.970470 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:42.970964 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:42.970971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:42.971321 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:42.971787 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.971800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.972671 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.972674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.973545 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:42.973554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:42.974075 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:42.974078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:42.974227 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:42.974836 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:42.974852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:42.976688 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:42.978598 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.978604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.978619 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.978885 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.979175 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:42.979179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:42.979181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:42.979330 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:42.979732 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:42.979741 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:42.979983 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 22:49:42.980168 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 22:49:42.980500 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260925 22:49:42.982585 7727 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37295
I20260925 22:49:42.982849 7727 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:42.983336 7727 heartbeater.cc:511] Master 127.4.219.190:37295 requested a full tablet report, sending...
I20260925 22:49:42.984408 7402 ts_manager.cc:194] Registered new tserver with Master: a9a8e86cf2cf4bafad8b038ef1ac503a (127.4.219.130:43597)
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:49:42.985248 7402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:60579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:43.003093 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:43.115417 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:43.115698 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:43.115741 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:43.119941 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:43.120016 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:43.120079 7732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:43.125571 7732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:43.126606 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:43.127568 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:43.137100 7740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:43.137161 7732 server_base.cc:1034] running on GCE node
W20260925 22:49:43.137176 7738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:43.137100 7737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:43.137782 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:43.138391 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:43.139561 7732 hybrid_clock.cc:648] HybridClock initialized: now 1790376583139543 us; error 32 us; skew 500 ppm
Sep 25 22:49:43 dist-test-slave-kb1g krb5kdc[7357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:43.142385 7732 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:43.143301 7732 webserver.cc:460] Webserver started at http://127.4.219.131:38959/ using document root <none> and password file <none>
I20260925 22:49:43.143915 7732 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:43.144007 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:43.144239 7732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:43.146344 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "b4ef273bcdd74a719e7d093f2ef30691"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-kb1g"
server_key: "db60f632028d9015cee73d4448339c85"
server_key_iv: "0de03341ce71cb0fde0bb207fd56d02e"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.146878 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ef273bcdd74a719e7d093f2ef30691"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-kb1g"
server_key: "db60f632028d9015cee73d4448339c85"
server_key_iv: "0de03341ce71cb0fde0bb207fd56d02e"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.150575 7732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:43.152917 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.154095 7732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 22:49:43.154223 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "b4ef273bcdd74a719e7d093f2ef30691"
format_stamp: "Formatted at 2026-09-25 22:49:43 on dist-test-slave-kb1g"
server_key: "db60f632028d9015cee73d4448339c85"
server_key_iv: "0de03341ce71cb0fde0bb207fd56d02e"
server_key_version: "encryptionkey@0"
I20260925 22:49:43.154328 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:43.179014 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:43.182719 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:43.182950 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:43.183596 7732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:43.184494 7732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:43.184592 7732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.184684 7732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:43.184736 7732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:43.193974 7732 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:41735
I20260925 22:49:43.193997 7860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:41735 every 8 connection(s)
I20260925 22:49:43.194911 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:43.199599 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7732
I20260925 22:49:43.199700 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:43.200023 4974 external_mini_cluster.cc:1428] Setting key f14adc1828a7ba3fe4cd176e6219b6af
Sep 25 22:49:43 dist-test-slave-kb1g krb5kdc[7357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376583, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:43.208797 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.196818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:32929 (local address 127.4.219.190:37295)
0925 22:49:43.196954 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.196958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.197809 (+ 851us) server_negotiation.cc:407] Connection header received
0925 22:49:43.199108 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.199111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.199161 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.199238 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:43.200556 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.201204 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.202681 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.202993 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.203078 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.205502 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:43.205505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:43.205507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:43.205531 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:43.206863 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:43.207317 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:43.207321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:43.207322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:43.207409 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:43.207705 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:43.207708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:43.207710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:43.207881 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:43.207959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:43.208503 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 22:49:43.208676 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:49:43.209621 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:43.197188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37295 (local address 127.4.219.131:32929)
0925 22:49:43.197647 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:43.197688 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:43.198894 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:43.199397 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:43.199404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:43.199841 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:43.200377 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.200392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.201388 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.201394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.202539 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:43.202552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:43.203128 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:43.203131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:43.203289 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:43.203836 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:43.203852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:43.205353 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:43.206961 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:43.206966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:43.206980 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:43.207220 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:43.207501 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:43.207503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:43.207504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:43.207626 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:43.207965 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:43.207970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:43.208255 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:49:43.208413 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 22:49:43.208741 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260925 22:49:43.210783 7861 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37295
I20260925 22:49:43.211021 7861 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:43.211515 7861 heartbeater.cc:511] Master 127.4.219.190:37295 requested a full tablet report, sending...
I20260925 22:49:43.212519 7402 ts_manager.cc:194] Registered new tserver with Master: b4ef273bcdd74a719e7d093f2ef30691 (127.4.219.131:41735)
I20260925 22:49:43.213330 7402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:32929
I20260925 22:49:43.213989 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:49:43.227295 7441 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:43.225830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.4.219.190:37295)
0925 22:49:43.226016 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:43.226020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:43.226090 (+ 70us) server_negotiation.cc:407] Connection header received
0925 22:49:43.226204 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:43.226207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:43.226250 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:43.226325 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:43.227151 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38076: BlockingRecv error: recv got EOF from 127.0.0.1:38076 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:49:43.228427 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7464
I20260925 22:49:43.233886 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7598
I20260925 22:49:43.239166 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7732
I20260925 22:49:43.244470 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7370
2026-09-25T22:49:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2146 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:43 dist-test-slave-kb1g krb5kdc[7873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:43 dist-test-slave-kb1g krb5kdc[7873](info): set up 2 sockets
Sep 25 22:49:43 dist-test-slave-kb1g krb5kdc[7873](info): commencing operation
krb5kdc: starting...
W20260925 22:49:44.312752 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:44.435506 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41211
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:44.546422 7886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:44.546685 7886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:44.546725 7886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:44.550791 7886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:44.550848 7886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:44.550866 7886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:44.550884 7886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:44.550899 7886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:44.556222 7886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:44.557331 7886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:44.558362 7886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:44.566880 7894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:44.566847 7892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:44.566851 7891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:44.566982 7886 server_base.cc:1034] running on GCE node
I20260925 22:49:44.567637 7886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:44.568804 7886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:44.569991 7886 hybrid_clock.cc:648] HybridClock initialized: now 1790376584569956 us; error 47 us; skew 500 ppm
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376584, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:44.576745 7886 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:44.577767 7886 webserver.cc:460] Webserver started at http://127.4.219.190:44141/ using document root <none> and password file <none>
I20260925 22:49:44.578337 7886 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:44.578406 7886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:44.578655 7886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:44.580781 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "9d071593b7d546e5914b9a49979aa73d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-kb1g"
server_key: "eec8dadb0c31a5fe89f97ad40bc6011b"
server_key_iv: "40ca992401366c4847c174af15e96839"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.581343 7886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "9d071593b7d546e5914b9a49979aa73d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-kb1g"
server_key: "eec8dadb0c31a5fe89f97ad40bc6011b"
server_key_iv: "40ca992401366c4847c174af15e96839"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.585156 7886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:49:44.587663 7901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:44.589075 7886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:44.589234 7886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "9d071593b7d546e5914b9a49979aa73d"
format_stamp: "Formatted at 2026-09-25 22:49:44 on dist-test-slave-kb1g"
server_key: "eec8dadb0c31a5fe89f97ad40bc6011b"
server_key_iv: "40ca992401366c4847c174af15e96839"
server_key_version: "encryptionkey@0"
I20260925 22:49:44.589359 7886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:44.608564 7886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:44.612506 7886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:44.612746 7886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:44.619902 7886 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41211
I20260925 22:49:44.620189 7953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41211 every 8 connection(s)
I20260925 22:49:44.621043 7886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:44.621711 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 7886
I20260925 22:49:44.621845 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:44.622144 4974 external_mini_cluster.cc:1428] Setting key c4e2f0f1261b8fd4a3d350fe21ec2b31
I20260925 22:49:44.624331 7954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:44.631078 7954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d: Bootstrap starting.
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:44.633920 7954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:44.634790 7954 log.cc:824] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:44.637050 7954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d: No bootstrap required, opened a new log
I20260925 22:49:44.637079 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.623446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.4.219.190:41211)
0925 22:49:44.623876 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.623894 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.623931 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:49:44.624669 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.624688 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.625019 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.625298 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.626191 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.627181 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.628418 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.628972 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.629104 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.631933 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.631940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.631955 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:44.631986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.633951 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:44.634389 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:44.634395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:44.634401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:44.634537 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:44.634797 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:44.634802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:44.634807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:44.635314 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.635456 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.635823 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.636101 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":147,"threads_started":1}
I20260925 22:49:44.639889 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:44.641050 7954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } }
I20260925 22:49:44.641252 7954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:44.641330 7954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d071593b7d546e5914b9a49979aa73d, State: Initialized, Role: FOLLOWER
I20260925 22:49:44.641774 7954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } }
I20260925 22:49:44.641930 7954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:44.642017 7954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:44.642158 7954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:44.643045 7954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } }
I20260925 22:49:44.643384 7954 leader_election.cc:302] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d071593b7d546e5914b9a49979aa73d; no voters:
I20260925 22:49:44.643718 7954 leader_election.cc:288] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:44.643786 7960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:44.644071 7960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [term 1 LEADER]: Becoming Leader. State: Replica: 9d071593b7d546e5914b9a49979aa73d, State: Running, Role: LEADER
I20260925 22:49:44.644412 7960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } }
I20260925 22:49:44.645090 7954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:44.645603 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d071593b7d546e5914b9a49979aa73d. Latest consensus state: current_term: 1 leader_uuid: "9d071593b7d546e5914b9a49979aa73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } } }
I20260925 22:49:44.645712 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:44.646426 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d071593b7d546e5914b9a49979aa73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d071593b7d546e5914b9a49979aa73d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41211 } } }
I20260925 22:49:44.646525 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:44.647138 7964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:44.650087 7964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:44.657226 7964 catalog_manager.cc:1347] Generated new cluster ID: 4745b0f1db514584b7091ae4f5a12124
I20260925 22:49:44.657301 7964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:44.662271 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:44.655005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.4.219.190:41211)
0925 22:49:44.655188 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:44.655196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:44.655966 (+ 770us) server_negotiation.cc:407] Connection header received
0925 22:49:44.656191 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:44.656197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:44.656261 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:44.656351 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:44.657240 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.658114 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.659510 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:44.659913 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:44.660000 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:44.661124 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:44.661128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:44.661130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:44.661739 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:44.661777 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:44.661798 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:44.661888 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:44.662058 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:49:44.662124 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:49:44.678505 7964 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:44.679462 7964 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:44.685030 7964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d071593b7d546e5914b9a49979aa73d: Generated new TSK 0
I20260925 22:49:44.685679 7964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:44.698347 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 7886
2026-09-25T22:49:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1451 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7987](info): set up 2 sockets
Sep 25 22:49:44 dist-test-slave-kb1g krb5kdc[7987](info): commencing operation
krb5kdc: starting...
W20260925 22:49:45.756654 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:45 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:45.876082 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35761
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:45.983490 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:45.983735 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:45.983773 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:45.987677 8000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:45.987740 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:45.987759 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:45.987776 8000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:45.987792 8000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:45.993032 8000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:45.994012 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:45.994953 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.003310 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.003310 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.003371 8000 server_base.cc:1034] running on GCE node
W20260925 22:49:46.003402 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.004065 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.005304 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.006507 8000 hybrid_clock.cc:648] HybridClock initialized: now 1790376586006477 us; error 40 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.009373 8000 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:46.010265 8000 webserver.cc:460] Webserver started at http://127.4.219.190:41945/ using document root <none> and password file <none>
I20260925 22:49:46.010867 8000 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.010959 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.011219 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.013341 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "4b5d10cba929465483675285a76d6600"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "0de56c9c2b52a9255c4f11c0cb697686"
server_key_iv: "2eb8902f2191015997c2bee9196c70df"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.013911 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "4b5d10cba929465483675285a76d6600"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "0de56c9c2b52a9255c4f11c0cb697686"
server_key_iv: "2eb8902f2191015997c2bee9196c70df"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.017500 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:46.019827 8015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.021067 8000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:46.021202 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "4b5d10cba929465483675285a76d6600"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "0de56c9c2b52a9255c4f11c0cb697686"
server_key_iv: "2eb8902f2191015997c2bee9196c70df"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.021308 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.032781 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.035385 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.035583 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.042680 8000 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:35761
I20260925 22:49:46.042999 8067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:35761 every 8 connection(s)
I20260925 22:49:46.043637 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:46.046360 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:46.051676 8068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: Bootstrap starting.
I20260925 22:49:46.051921 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8000
I20260925 22:49:46.052039 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:46.052310 4974 external_mini_cluster.cc:1428] Setting key 27cf46b60178830f76653beae1435cac
I20260925 22:49:46.054535 8068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:46.055397 8068 log.cc:824] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:46.057615 8068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: No bootstrap required, opened a new log
I20260925 22:49:46.061457 8068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } }
I20260925 22:49:46.061663 8068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:46.061722 8068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b5d10cba929465483675285a76d6600, State: Initialized, Role: FOLLOWER
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:46.062168 8068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } }
I20260925 22:49:46.062299 8068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:46.062397 8068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:46.062538 8068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:46.063443 8068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } }
I20260925 22:49:46.063798 8068 leader_election.cc:302] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b5d10cba929465483675285a76d6600; no voters:
I20260925 22:49:46.064128 8068 leader_election.cc:288] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:46.064214 8073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:46.064421 8073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [term 1 LEADER]: Becoming Leader. State: Replica: 4b5d10cba929465483675285a76d6600, State: Running, Role: LEADER
I20260925 22:49:46.064720 8073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } }
I20260925 22:49:46.065438 8068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:46.066284 8075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b5d10cba929465483675285a76d6600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } } }
I20260925 22:49:46.066419 8075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:46.066783 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b5d10cba929465483675285a76d6600. Latest consensus state: current_term: 1 leader_uuid: "4b5d10cba929465483675285a76d6600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b5d10cba929465483675285a76d6600" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35761 } } }
I20260925 22:49:46.066872 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:46.068146 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.053728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.4.219.190:35761)
0925 22:49:46.054152 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.054170 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.054210 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:49:46.055075 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.055096 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.055521 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.055890 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.056705 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.058148 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.058975 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.059666 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.059828 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.062529 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.062536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.062553 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.062583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.064646 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.065150 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.065155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.065159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.065261 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.065586 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.065589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.065591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.066145 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.066503 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.066719 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.067130 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
W20260925 22:49:46.070331 8088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:46.070406 8088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:46.070461 8082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:46.071281 8082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:46.076571 8082 catalog_manager.cc:1347] Generated new cluster ID: e38ff9d8b4694d8d96091915a25b072e
I20260925 22:49:46.076635 8082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:46.100678 8082 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:46.101967 8082 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:46.110373 8082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b5d10cba929465483675285a76d6600: Generated new TSK 0
I20260925 22:49:46.111202 8082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:46.139539 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.250830 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.251111 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.251155 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.255241 8094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.255299 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.255360 8094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:46.260668 8094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:46.261664 8094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.262614 8094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.272089 8099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.272068 8100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.272116 8102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.272461 8094 server_base.cc:1034] running on GCE node
I20260925 22:49:46.272897 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.273540 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.274699 8094 hybrid_clock.cc:648] HybridClock initialized: now 1790376586274672 us; error 37 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.277498 8094 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:46.278393 8094 webserver.cc:460] Webserver started at http://127.4.219.129:39719/ using document root <none> and password file <none>
I20260925 22:49:46.278991 8094 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.279088 8094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.279336 8094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.281556 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "ae2b0c96dd0b4fce944390ca73b040da"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "3f0d3fcdebb60c4c53db38240bfed889"
server_key_iv: "2b653dad6e6485aa65ea15037f17643c"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.282162 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "ae2b0c96dd0b4fce944390ca73b040da"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "3f0d3fcdebb60c4c53db38240bfed889"
server_key_iv: "2b653dad6e6485aa65ea15037f17643c"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.286239 8094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:46.289084 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.290343 8094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:46.290493 8094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "ae2b0c96dd0b4fce944390ca73b040da"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "3f0d3fcdebb60c4c53db38240bfed889"
server_key_iv: "2b653dad6e6485aa65ea15037f17643c"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.290616 8094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.306730 8094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.309895 8094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.310099 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.310729 8094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.311594 8094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.311640 8094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.311705 8094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.311735 8094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.320796 8094 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:46765
I20260925 22:49:46.320858 8222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:46765 every 8 connection(s)
I20260925 22:49:46.321703 8094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:46.325687 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8094
I20260925 22:49:46.325870 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:46.326151 4974 external_mini_cluster.cc:1428] Setting key 152715e7c19c266679f1120e21d4f2a3
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:46.336681 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.323608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:41515 (local address 127.4.219.190:35761)
0925 22:49:46.323746 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.323750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.324673 (+ 923us) server_negotiation.cc:407] Connection header received
0925 22:49:46.326295 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.326299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.326355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.326435 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.327792 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.328334 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.330115 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.330370 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.330461 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.332984 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.332988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.332990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.333014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.334475 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.335015 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.335023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.335024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.335098 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.335457 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.335460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.335462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.335685 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.335788 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.336368 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.336494 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:49:46.337590 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.324006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:35761 (local address 127.4.219.129:41515)
0925 22:49:46.324473 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.324516 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.326079 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.326614 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.326622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.327069 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.327646 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.327660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.328465 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.328468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.329824 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.329839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.330516 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.330519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.330698 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.331300 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.331316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.332805 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.334607 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.334613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.334630 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.334899 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.335198 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.335201 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.335203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.335354 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.335787 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.335796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.336057 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.336311 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.336692 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:46.338946 8223 heartbeater.cc:347] Connected to a master server at 127.4.219.190:35761
I20260925 22:49:46.339214 8223 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:46.339727 8223 heartbeater.cc:511] Master 127.4.219.190:35761 requested a full tablet report, sending...
I20260925 22:49:46.341483 8032 ts_manager.cc:194] Registered new tserver with Master: ae2b0c96dd0b4fce944390ca73b040da (127.4.219.129:46765)
I20260925 22:49:46.343403 8032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:41515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:46.353837 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.477524 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.477816 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.477865 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.482365 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.482424 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.482525 8228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:46.488291 8228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:46.489449 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.490540 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.501040 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.501140 8228 server_base.cc:1034] running on GCE node
W20260925 22:49:46.501158 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.501053 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.501847 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.502481 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.503693 8228 hybrid_clock.cc:648] HybridClock initialized: now 1790376586503673 us; error 33 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.506665 8228 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:46.507606 8228 webserver.cc:460] Webserver started at http://127.4.219.130:40421/ using document root <none> and password file <none>
I20260925 22:49:46.508201 8228 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.508309 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.508615 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.510838 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "145df10e12cb4486ab73f6771a8b2fa2"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "6efc33c3832b069b19d92e9f023220a3"
server_key_iv: "e38b55d7be94f13be40338d48f9fbfa5"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.511456 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "145df10e12cb4486ab73f6771a8b2fa2"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "6efc33c3832b069b19d92e9f023220a3"
server_key_iv: "e38b55d7be94f13be40338d48f9fbfa5"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.515384 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:49:46.518029 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.519292 8228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:46.519411 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "145df10e12cb4486ab73f6771a8b2fa2"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "6efc33c3832b069b19d92e9f023220a3"
server_key_iv: "e38b55d7be94f13be40338d48f9fbfa5"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.519497 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.540606 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.543579 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.543774 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.544365 8228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.545241 8228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.545288 8228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.545331 8228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.545368 8228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.554272 8228 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:45203
I20260925 22:49:46.554299 8356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:45203 every 8 connection(s)
I20260925 22:49:46.555173 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:46.560216 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8228
I20260925 22:49:46.560379 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:46.560714 4974 external_mini_cluster.cc:1428] Setting key 44d619e9a9012cb133f304b528180a89
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:49:46.570019 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.557094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:53999 (local address 127.4.219.190:35761)
0925 22:49:46.557249 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.557253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.558109 (+ 856us) server_negotiation.cc:407] Connection header received
0925 22:49:46.559417 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.559420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.559465 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.559539 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.560821 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.561504 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.563094 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.563340 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.563421 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.566238 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.566243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.566249 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.566276 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.568017 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.568534 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.568539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.568560 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.568646 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.568980 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.568984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.568986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.569158 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.569256 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.569731 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.569852 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:49:46.570798 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.557448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:35761 (local address 127.4.219.130:53999)
0925 22:49:46.557961 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.558003 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.559227 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.559701 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.559707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.560058 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.560673 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.560690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.561626 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.561629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.562961 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.562975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.563491 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.563498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.563729 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.564406 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.564421 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.566073 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.568126 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.568132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.568145 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.568432 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.568737 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.568741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.568743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.568886 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.569273 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.569279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.569514 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.569694 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.569995 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20260925 22:49:46.571870 8357 heartbeater.cc:347] Connected to a master server at 127.4.219.190:35761
I20260925 22:49:46.572126 8357 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:49:46.572687 8357 heartbeater.cc:511] Master 127.4.219.190:35761 requested a full tablet report, sending...
I20260925 22:49:46.573719 8032 ts_manager.cc:194] Registered new tserver with Master: 145df10e12cb4486ab73f6771a8b2fa2 (127.4.219.130:45203)
I20260925 22:49:46.574576 8032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:53999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:46.587356 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:46.696012 8362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:46.696245 8362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:46.696291 8362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:46.700192 8362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:46.700239 8362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:46.700295 8362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:46.705410 8362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:35761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:46.706367 8362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:46.707264 8362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:46.716902 8368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.716902 8367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:46.717026 8370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:46.717281 8362 server_base.cc:1034] running on GCE node
I20260925 22:49:46.717720 8362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:46.718330 8362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:46.719507 8362 hybrid_clock.cc:648] HybridClock initialized: now 1790376586719486 us; error 44 us; skew 500 ppm
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:46.722332 8362 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:46.723371 8362 webserver.cc:460] Webserver started at http://127.4.219.131:43745/ using document root <none> and password file <none>
I20260925 22:49:46.723968 8362 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:46.724030 8362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:46.724257 8362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:46.726411 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "18a0e48b2db24821af61865addcbb15f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "e686b4925d01df1da492236a9adb4a10"
server_key_iv: "fb17b1b9ac31e325c55f4a5540d10f56"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.726891 8362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "18a0e48b2db24821af61865addcbb15f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "e686b4925d01df1da492236a9adb4a10"
server_key_iv: "fb17b1b9ac31e325c55f4a5540d10f56"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.730441 8362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:46.732749 8377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.733934 8362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:46.734025 8362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "18a0e48b2db24821af61865addcbb15f"
format_stamp: "Formatted at 2026-09-25 22:49:46 on dist-test-slave-kb1g"
server_key: "e686b4925d01df1da492236a9adb4a10"
server_key_iv: "fb17b1b9ac31e325c55f4a5540d10f56"
server_key_version: "encryptionkey@0"
I20260925 22:49:46.734102 8362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:46.751127 8362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:46.754354 8362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:46.754495 8362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:46.755015 8362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:46.755756 8362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:46.755807 8362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.755848 8362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:46.755863 8362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:46.765093 8362 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:44131
I20260925 22:49:46.765111 8490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:44131 every 8 connection(s)
I20260925 22:49:46.766073 8362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:46.773412 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8362
I20260925 22:49:46.773561 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:46.773896 4974 external_mini_cluster.cc:1428] Setting key ccac9eb8772bf5378eb80940b0f1603a
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:46.781039 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.768067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:49301 (local address 127.4.219.190:35761)
0925 22:49:46.768217 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.768222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.769112 (+ 890us) server_negotiation.cc:407] Connection header received
0925 22:49:46.770616 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.770619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.770683 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.770813 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.772101 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.772813 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.774543 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.774798 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.774925 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.777694 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.777697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.777699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.777722 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.779140 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.779623 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.779625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.779627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.779696 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.780005 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.780007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.780008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.780176 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.780248 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.780832 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.780923 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:49:46.781950 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.768403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:35761 (local address 127.4.219.131:49301)
0925 22:49:46.768939 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:46.768986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:46.770383 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:46.770993 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:46.771000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:46.771424 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:46.771942 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.771957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.772947 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.772952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.774406 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:46.774415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.774963 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.774967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:46.775181 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.775870 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:46.775895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:46.777537 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:46.779255 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.779260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.779272 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.779534 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.779775 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:46.779778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:46.779780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:46.779924 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:46.780273 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:46.780279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:46.780528 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 22:49:46.780789 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 22:49:46.781054 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":181,"threads_started":1}
I20260925 22:49:46.783175 8491 heartbeater.cc:347] Connected to a master server at 127.4.219.190:35761
I20260925 22:49:46.783408 8491 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:46.783850 8491 heartbeater.cc:511] Master 127.4.219.190:35761 requested a full tablet report, sending...
I20260925 22:49:46.784803 8032 ts_manager.cc:194] Registered new tserver with Master: 18a0e48b2db24821af61865addcbb15f (127.4.219.131:44131)
I20260925 22:49:46.785367 8032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:49301
I20260925 22:49:46.787887 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[7987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376586, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:46.813658 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:46.805103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.4.219.190:35761)
0925 22:49:46.805303 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:46.805308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:46.805419 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:49:46.805592 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:46.805595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:46.805644 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:46.805723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:46.806508 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.807027 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.807856 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:46.808088 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:46.808198 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:46.810536 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:46.810539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:46.810541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:46.810564 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:46.811991 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.812375 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.812377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.812379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.812453 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:46.812744 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:46.812747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:46.812748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:46.812982 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:46.813083 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:46.813376 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 22:49:46.813491 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 22:49:46.814121 8032 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44260
I20260925 22:49:46.815194 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8094
I20260925 22:49:46.820858 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8228
I20260925 22:49:46.826081 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8362
I20260925 22:49:46.831207 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8000
2026-09-25T22:49:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2134 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[8503](info): set up 2 sockets
Sep 25 22:49:46 dist-test-slave-kb1g krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260925 22:49:47.868757 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:47 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:47.992127 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39077
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:48.102540 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:48.102892 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:48.102975 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:48.107136 8516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:48.107230 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:48.107271 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:48.107329 8516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:48.107370 8516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:48.112759 8516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:48.113879 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:48.114943 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:48.123610 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.123600 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.123761 8516 server_base.cc:1034] running on GCE node
W20260925 22:49:48.123596 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.124501 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:48.125622 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:48.126787 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790376588126772 us; error 31 us; skew 500 ppm
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:48.129667 8516 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:48.130615 8516 webserver.cc:460] Webserver started at http://127.4.219.190:41955/ using document root <none> and password file <none>
I20260925 22:49:48.131194 8516 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:48.131254 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:48.131500 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:48.133685 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "8671b05e91224d749fa930041930546d"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "2d28f729d614ed0b9233771c2cb8be9f"
server_key_iv: "99165b64669edca1b7260e8ab40c3b28"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.134294 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "8671b05e91224d749fa930041930546d"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "2d28f729d614ed0b9233771c2cb8be9f"
server_key_iv: "99165b64669edca1b7260e8ab40c3b28"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.138329 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:48.140996 8531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.142333 8516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:48.142498 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "8671b05e91224d749fa930041930546d"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "2d28f729d614ed0b9233771c2cb8be9f"
server_key_iv: "99165b64669edca1b7260e8ab40c3b28"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.142632 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:48.157219 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:48.160750 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:48.160960 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:48.168180 8516 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39077
I20260925 22:49:48.168191 8583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39077 every 8 connection(s)
I20260925 22:49:48.169204 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:48.172119 8584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:48.177874 8584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d: Bootstrap starting.
I20260925 22:49:48.179337 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8516
I20260925 22:49:48.179469 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:48.179733 4974 external_mini_cluster.cc:1428] Setting key 0702dd03fc3ec721b8195d36069294b5
I20260925 22:49:48.180935 8584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:48.181807 8584 log.cc:824] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:48.183928 8584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d: No bootstrap required, opened a new log
I20260925 22:49:48.187500 8584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } }
I20260925 22:49:48.187770 8584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:48.187861 8584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8671b05e91224d749fa930041930546d, State: Initialized, Role: FOLLOWER
I20260925 22:49:48.188309 8584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } }
I20260925 22:49:48.188462 8584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:48.188566 8584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:48.188690 8584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 0 FOLLOWER]: Advancing to term 1
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:48.189677 8584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } }
I20260925 22:49:48.190033 8584 leader_election.cc:302] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8671b05e91224d749fa930041930546d; no voters:
I20260925 22:49:48.190382 8584 leader_election.cc:288] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:48.190460 8589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:48.190654 8589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [term 1 LEADER]: Becoming Leader. State: Replica: 8671b05e91224d749fa930041930546d, State: Running, Role: LEADER
I20260925 22:49:48.190969 8589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } }
I20260925 22:49:48.191738 8584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:48.192721 8590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8671b05e91224d749fa930041930546d. Latest consensus state: current_term: 1 leader_uuid: "8671b05e91224d749fa930041930546d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } } }
I20260925 22:49:48.193104 8590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:48.192999 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8671b05e91224d749fa930041930546d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8671b05e91224d749fa930041930546d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39077 } } }
I20260925 22:49:48.193454 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:48.193742 8598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:48.195562 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.181066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.4.219.190:39077)
0925 22:49:48.181544 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.181560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.181597 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:49:48.182442 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.182470 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.182864 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.183228 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.184031 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.185134 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.186023 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.186508 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.186687 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.189486 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.189497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.189511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.189538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.191626 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.192111 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.192117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.192123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.192300 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.192710 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.192717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.192719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.193344 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.193555 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.194188 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.194590 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":154,"threads_started":1}
I20260925 22:49:48.197646 8598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:48.203738 8598 catalog_manager.cc:1347] Generated new cluster ID: 13311ee39930463486b8e8de45f59972
I20260925 22:49:48.203861 8598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:48.229548 8598 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:48.230868 8598 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:48.243232 8598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8671b05e91224d749fa930041930546d: Generated new TSK 0
I20260925 22:49:48.244119 8598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:48.282022 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:48.419674 8610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:48.419937 8610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:48.420042 8610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:48.426374 8610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:48.426438 8610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:48.426538 8610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:48.431895 8610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:48.433022 8610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:48.434060 8610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:48.444792 8618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.444825 8616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.444861 8610 server_base.cc:1034] running on GCE node
W20260925 22:49:48.444801 8615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.445451 8610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:48.446187 8610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:48.447369 8610 hybrid_clock.cc:648] HybridClock initialized: now 1790376588447331 us; error 48 us; skew 500 ppm
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:48.450215 8610 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:48.451176 8610 webserver.cc:460] Webserver started at http://127.4.219.129:38405/ using document root <none> and password file <none>
I20260925 22:49:48.451745 8610 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:48.451834 8610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:48.452069 8610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:48.454280 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "abeefdd443ac4e3588f0c7dc7b0b0e8a"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "b5f2bf80f3ea957b9db3c7e0dcef4ba8"
server_key_iv: "db513c0ee7ed4b4d088a55b72598a2ac"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.454838 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "abeefdd443ac4e3588f0c7dc7b0b0e8a"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "b5f2bf80f3ea957b9db3c7e0dcef4ba8"
server_key_iv: "db513c0ee7ed4b4d088a55b72598a2ac"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.458532 8610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 22:49:48.461031 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.462405 8610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:48.462522 8610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "abeefdd443ac4e3588f0c7dc7b0b0e8a"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "b5f2bf80f3ea957b9db3c7e0dcef4ba8"
server_key_iv: "db513c0ee7ed4b4d088a55b72598a2ac"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.462633 8610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:48.482364 8610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:48.485311 8610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:48.485469 8610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:48.486052 8610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:48.486809 8610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:48.486867 8610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.486930 8610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:48.486946 8610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.495934 8610 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:33095
I20260925 22:49:48.495956 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:33095 every 8 connection(s)
I20260925 22:49:48.496881 8610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:48.499670 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8610
I20260925 22:49:48.499784 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:48.500082 4974 external_mini_cluster.cc:1428] Setting key 9fd895aad9c0bf51b799edcaf6c56182
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:48.512411 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.498682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:35745 (local address 127.4.219.190:39077)
0925 22:49:48.498834 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.498838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.499755 (+ 917us) server_negotiation.cc:407] Connection header received
0925 22:49:48.501472 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.501476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.501526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.501619 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.503334 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.503879 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.505406 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.505636 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.505730 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.508708 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.508712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.508714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.508737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.510177 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.510788 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.510790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.510792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.510863 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.511240 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.511244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.511245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.511461 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.511568 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.512081 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.512214 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:48.513450 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.499092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39077 (local address 127.4.219.129:35745)
0925 22:49:48.499565 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:48.499627 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:48.501236 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:48.501868 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:48.501878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:48.502406 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:48.503133 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.503156 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.504019 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.504022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.505268 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.505283 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.505805 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.505812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.506043 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.506713 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:48.506730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:48.508480 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:48.510325 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.510331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.510347 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.510668 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.510974 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.510978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.510980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.511122 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.511564 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:48.511570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:48.511831 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 22:49:48.512028 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 22:49:48.512376 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":98,"threads_started":1}
I20260925 22:49:48.514839 8739 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39077
I20260925 22:49:48.515125 8739 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:48.515758 8739 heartbeater.cc:511] Master 127.4.219.190:39077 requested a full tablet report, sending...
I20260925 22:49:48.517366 8548 ts_manager.cc:194] Registered new tserver with Master: abeefdd443ac4e3588f0c7dc7b0b0e8a (127.4.219.129:33095)
I20260925 22:49:48.518491 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:35745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:48.529826 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:48.639816 8744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:48.640070 8744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:48.640111 8744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:48.644147 8744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:48.644213 8744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:48.644307 8744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:48.649748 8744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:48.650844 8744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:48.651930 8744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:48.661936 8744 server_base.cc:1034] running on GCE node
W20260925 22:49:48.663931 8749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.663995 8752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.664072 8750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.664501 8744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:48.665227 8744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:48.666414 8744 hybrid_clock.cc:648] HybridClock initialized: now 1790376588666379 us; error 48 us; skew 500 ppm
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:48.669247 8744 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:48.670143 8744 webserver.cc:460] Webserver started at http://127.4.219.130:44219/ using document root <none> and password file <none>
I20260925 22:49:48.670692 8744 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:48.670750 8744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:48.670921 8744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:48.672991 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "fea50219fa464182aedbfd2d9263800c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "f847f00ef8b22692a61ee0e4585f8aea"
server_key_iv: "2989206986951f7bd867d6ac698d1817"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.673491 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "fea50219fa464182aedbfd2d9263800c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "f847f00ef8b22692a61ee0e4585f8aea"
server_key_iv: "2989206986951f7bd867d6ac698d1817"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.677150 8744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:48.679385 8759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.680624 8744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:48.680747 8744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "fea50219fa464182aedbfd2d9263800c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "f847f00ef8b22692a61ee0e4585f8aea"
server_key_iv: "2989206986951f7bd867d6ac698d1817"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.680855 8744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:48.691656 8744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:48.694958 8744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:48.695155 8744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:48.695734 8744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:48.696671 8744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:48.696743 8744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.696834 8744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:48.696878 8744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.705391 8744 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:38577
I20260925 22:49:48.705436 8872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:38577 every 8 connection(s)
I20260925 22:49:48.706300 8744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:48.706483 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8744
I20260925 22:49:48.706600 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:48.706939 4974 external_mini_cluster.cc:1428] Setting key d26dda24d2980cb88c34cace7275a0c0
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:48.725226 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.708328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:57043 (local address 127.4.219.190:39077)
0925 22:49:48.708454 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.708458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.709518 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 22:49:48.711410 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.711413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.711458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.711534 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.713171 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.713685 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.715502 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.715698 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.715772 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.721140 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.721144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.721146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.721170 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.722664 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.723336 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.723343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.723348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.723446 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.723795 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.723802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.723807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.724076 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.724190 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.724825 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.724974 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:49:48.725937 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.708863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39077 (local address 127.4.219.130:57043)
0925 22:49:48.709378 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:48.709423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:48.711149 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:48.711745 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:48.711757 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:48.712267 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:48.713001 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.713026 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.713819 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.713824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.715385 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.715395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.715848 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.715855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.716071 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.716766 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:48.716791 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:48.720942 (+ 4151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:48.722884 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.722889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.722903 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.723163 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.723537 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.723539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.723540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.723676 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.724210 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:48.724216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:48.724476 (+ 260us) client_negotiation.cc:769] Sending connection context
0925 22:49:48.724710 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:49:48.724999 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260925 22:49:48.727416 8873 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39077
I20260925 22:49:48.727695 8873 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:48.728333 8873 heartbeater.cc:511] Master 127.4.219.190:39077 requested a full tablet report, sending...
I20260925 22:49:48.729539 8548 ts_manager.cc:194] Registered new tserver with Master: fea50219fa464182aedbfd2d9263800c (127.4.219.130:38577)
I20260925 22:49:48.730242 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:57043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:48.735675 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:48.848727 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:48.849041 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:48.849088 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:48.853435 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:48.853513 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:48.853607 8878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:48.859213 8878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:48.860347 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:48.861469 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:48.871824 8884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.871775 8883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:48.871771 8886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:48.872153 8878 server_base.cc:1034] running on GCE node
I20260925 22:49:48.872699 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:48.873311 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:48.874529 8878 hybrid_clock.cc:648] HybridClock initialized: now 1790376588874514 us; error 34 us; skew 500 ppm
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:48.877846 8878 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:48.878832 8878 webserver.cc:460] Webserver started at http://127.4.219.131:35423/ using document root <none> and password file <none>
I20260925 22:49:48.879457 8878 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:48.879554 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:48.879794 8878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:48.882071 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "32ac0d273ab043ba848e47d88c09db0c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "a3109b9a433aba0de02eef193b9f2747"
server_key_iv: "3e54fe6a65114a2744b21f53d3cdceec"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.882735 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "32ac0d273ab043ba848e47d88c09db0c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "a3109b9a433aba0de02eef193b9f2747"
server_key_iv: "3e54fe6a65114a2744b21f53d3cdceec"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.887073 8878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:48.889835 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.891193 8878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:48.891399 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "32ac0d273ab043ba848e47d88c09db0c"
format_stamp: "Formatted at 2026-09-25 22:49:48 on dist-test-slave-kb1g"
server_key: "a3109b9a433aba0de02eef193b9f2747"
server_key_iv: "3e54fe6a65114a2744b21f53d3cdceec"
server_key_version: "encryptionkey@0"
I20260925 22:49:48.891533 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:48.915802 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:48.919440 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:48.919662 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:48.920306 8878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:48.921319 8878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:48.921401 8878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.921492 8878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:48.921542 8878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:48.930585 8878 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:37557
I20260925 22:49:48.930596 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:37557 every 8 connection(s)
I20260925 22:49:48.931674 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:48.932716 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 8878
I20260925 22:49:48.932883 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:48.933212 4974 external_mini_cluster.cc:1428] Setting key 893ab1b069109027ca04c53311b50d6d
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:48.946477 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.933780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:43507 (local address 127.4.219.190:39077)
0925 22:49:48.933950 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.933954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.934988 (+ 1034us) server_negotiation.cc:407] Connection header received
0925 22:49:48.936673 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.936676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.936728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.936794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.937987 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.938520 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.939732 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.939978 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.940070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.942672 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.942676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.942678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.942704 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.944208 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.944837 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.944840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.944841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.944918 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.945229 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.945232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.945233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.945414 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.945489 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.946170 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.946306 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:49:48.947577 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.934310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39077 (local address 127.4.219.131:43507)
0925 22:49:48.934817 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:48.934878 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:48.936406 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:48.936935 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:48.936942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:48.937329 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:48.937837 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.937851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.938648 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.938651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.939574 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:48.939583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.940111 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.940114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:48.940254 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.940972 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:48.940989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:48.942504 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:48.944360 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.944365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.944383 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.944748 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.945003 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:48.945005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:48.945006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:48.945144 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:48.945527 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:48.945532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:48.945858 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 22:49:48.946109 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 22:49:48.946533 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":98,"threads_started":1}
I20260925 22:49:48.948673 9007 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39077
I20260925 22:49:48.948916 9007 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:48.949419 9007 heartbeater.cc:511] Master 127.4.219.190:39077 requested a full tablet report, sending...
I20260925 22:49:48.950379 8548 ts_manager.cc:194] Registered new tserver with Master: 32ac0d273ab043ba848e47d88c09db0c (127.4.219.131:37557)
I20260925 22:49:48.950910 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:43507
I20260925 22:49:48.959487 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:48.984329 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.975477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.4.219.129:33095)
0925 22:49:48.975766 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.975786 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.975809 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:49:48.975859 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.975864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.976052 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.976170 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.976882 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.977636 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.978424 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.978775 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.978860 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.981114 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.981121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.981134 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.981156 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.982770 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.983146 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.983151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.983152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.983240 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.983460 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.983462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.983463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.983676 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.983973 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.984063 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.984182 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
Sep 25 22:49:48 dist-test-slave-kb1g krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:48.995590 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:48.987212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.4.219.190:39077)
0925 22:49:48.987350 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:48.987357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:48.987378 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:49:48.987532 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:48.987535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:48.987586 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:48.987667 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:48.988258 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.988848 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.989606 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:48.989825 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:48.989930 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:48.992187 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:48.992190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:48.992193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:48.992223 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:48.993690 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.994176 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.994180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.994182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.994268 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:48.994539 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:48.994544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:48.994546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:48.994809 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:48.994985 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:48.995273 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:49:48.995404 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:49:48.996095 8548 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46384
I20260925 22:49:48.996717 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8610
I20260925 22:49:49.002692 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8744
I20260925 22:49:49.007754 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8878
I20260925 22:49:49.013195 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 8516
2026-09-25T22:49:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2182 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:49 dist-test-slave-kb1g krb5kdc[9019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:49 dist-test-slave-kb1g krb5kdc[9019](info): set up 2 sockets
Sep 25 22:49:49 dist-test-slave-kb1g krb5kdc[9019](info): commencing operation
krb5kdc: starting...
W20260925 22:49:50.056835 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:50.184324 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44307
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:50.297899 9032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.298255 9032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.298336 9032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.302527 9032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:50.302640 9032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.302676 9032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.302716 9032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:50.302753 9032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:50.308358 9032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:50.309612 9032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.310724 9032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.319759 9038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.319840 9037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.319825 9032 server_base.cc:1034] running on GCE node
W20260925 22:49:50.319742 9040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.320609 9032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.321884 9032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.323096 9032 hybrid_clock.cc:648] HybridClock initialized: now 1790376590323056 us; error 51 us; skew 500 ppm
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:50.335476 9032 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:50.336380 9032 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.336484 9032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.336786 9032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.339015 9032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "b016f9f1a84d49d28c44f34217c0aad8"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "95b6ef31f8bceea06b8b51336643fa8c"
server_key_iv: "7c6ff8c19a3b092071ba73661e9c65bf"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.339612 9032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "b016f9f1a84d49d28c44f34217c0aad8"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "95b6ef31f8bceea06b8b51336643fa8c"
server_key_iv: "7c6ff8c19a3b092071ba73661e9c65bf"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.343634 9032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:49:50.346071 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.347352 9032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:50.347509 9032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "b016f9f1a84d49d28c44f34217c0aad8"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "95b6ef31f8bceea06b8b51336643fa8c"
server_key_iv: "7c6ff8c19a3b092071ba73661e9c65bf"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.347631 9032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.375705 9032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.378921 9032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.379153 9032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.386559 9032 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44307
I20260925 22:49:50.386896 9098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44307 every 8 connection(s)
I20260925 22:49:50.387292 9032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:50.390376 9099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:50.391099 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9032
I20260925 22:49:50.391256 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:50.391551 4974 external_mini_cluster.cc:1428] Setting key bf9cc51bd296c48a41a17b194c69d0a6
I20260925 22:49:50.397707 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: Bootstrap starting.
I20260925 22:49:50.400610 9099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:50.401425 9099 log.cc:824] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:50.403899 9099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: No bootstrap required, opened a new log
I20260925 22:49:50.408571 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.393181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.4.219.190:44307)
0925 22:49:50.393648 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.393669 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.393716 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:49:50.394633 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.394663 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.395123 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.395551 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.396379 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.398019 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.399006 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.399742 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.399908 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.403084 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.403090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.403107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:50.403150 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.405276 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.405747 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.405753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.405758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.405918 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.406175 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.406178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.406182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.406789 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.406993 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.407204 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.407517 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
I20260925 22:49:50.408725 9099 raft_consensus.cc:348] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } }
I20260925 22:49:50.409021 9099 raft_consensus.cc:374] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:50.409082 9099 raft_consensus.cc:729] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b016f9f1a84d49d28c44f34217c0aad8, State: Initialized, Role: FOLLOWER
I20260925 22:49:50.409570 9099 consensus_queue.cc:260] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } }
I20260925 22:49:50.409727 9099 raft_consensus.cc:388] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:50.409793 9099 raft_consensus.cc:482] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:50.409897 9099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:50.410902 9099 raft_consensus.cc:504] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } }
I20260925 22:49:50.411252 9099 leader_election.cc:302] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b016f9f1a84d49d28c44f34217c0aad8; no voters:
I20260925 22:49:50.411577 9099 leader_election.cc:288] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:50.411703 9104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:50.411949 9104 raft_consensus.cc:686] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [term 1 LEADER]: Becoming Leader. State: Replica: b016f9f1a84d49d28c44f34217c0aad8, State: Running, Role: LEADER
I20260925 22:49:50.412278 9104 consensus_queue.cc:237] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } }
I20260925 22:49:50.412935 9099 sys_catalog.cc:564] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:50.414150 9105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b016f9f1a84d49d28c44f34217c0aad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } } }
I20260925 22:49:50.414166 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b016f9f1a84d49d28c44f34217c0aad8. Latest consensus state: current_term: 1 leader_uuid: "b016f9f1a84d49d28c44f34217c0aad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b016f9f1a84d49d28c44f34217c0aad8" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44307 } } }
I20260925 22:49:50.414392 9105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:50.414419 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:50.414816 9113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:50.418061 9113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:50.424758 9113 catalog_manager.cc:1347] Generated new cluster ID: 54c7d3b0d03344898968bab79d73c58b
I20260925 22:49:50.424854 9113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:50.433589 9113 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:50.434660 9113 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:50.443004 9113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: Generated new TSK 0
I20260925 22:49:50.443849 9113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:50.476001 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:50.592978 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.593262 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.593309 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.597460 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.597537 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.597635 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:50.603343 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:50.604504 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.605660 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.615607 9131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.615869 9125 server_base.cc:1034] running on GCE node
W20260925 22:49:50.615648 9133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.615592 9130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.616498 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.617273 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.618482 9125 hybrid_clock.cc:648] HybridClock initialized: now 1790376590618458 us; error 36 us; skew 500 ppm
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:50.621619 9125 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:50.622493 9125 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.622591 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.622844 9125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.625166 9125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "2b684fa3b3034cb1aaaaa74900da3b7a"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "6a30e375786a18a8552e3bde8c31df23"
server_key_iv: "576a683cafccbb341f2b11bc2f0786fb"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.625768 9125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "2b684fa3b3034cb1aaaaa74900da3b7a"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "6a30e375786a18a8552e3bde8c31df23"
server_key_iv: "576a683cafccbb341f2b11bc2f0786fb"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.629740 9125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:50.632452 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.633903 9125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:50.634068 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "2b684fa3b3034cb1aaaaa74900da3b7a"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "6a30e375786a18a8552e3bde8c31df23"
server_key_iv: "576a683cafccbb341f2b11bc2f0786fb"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.634203 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.665920 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.669485 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.669697 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.670318 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:50.671178 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:50.671255 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.671329 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:50.671386 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.680395 9125 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:42511
I20260925 22:49:50.680413 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:42511 every 8 connection(s)
I20260925 22:49:50.681064 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:50.683269 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9125
I20260925 22:49:50.683429 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:50.683813 4974 external_mini_cluster.cc:1428] Setting key 401ac95f524032827f0411f4a61bf509
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:50.697111 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.682862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:56849 (local address 127.4.219.190:44307)
0925 22:49:50.683032 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.683037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.684077 (+ 1040us) server_negotiation.cc:407] Connection header received
0925 22:49:50.685776 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.685779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.685819 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.685878 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.687410 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.688258 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.690124 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.690357 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.690438 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.693440 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.693446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.693451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:50.693487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.694919 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.695485 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.695488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.695489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.695564 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.695856 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.695859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.695860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.696038 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.696112 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.696830 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.696976 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:49:50.698418 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.683300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44307 (local address 127.4.219.129:56849)
0925 22:49:50.683894 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.683956 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.685546 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.686064 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.686076 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.686611 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:50.687261 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.687276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.688392 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.688398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.690001 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.690013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.690527 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.690533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.690785 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.691485 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:50.691514 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:50.693261 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:50.695058 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.695063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.695077 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.695382 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.695640 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.695642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.695643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.695775 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.696170 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:50.696176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:50.696474 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.696769 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.697212 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
I20260925 22:49:50.699848 9253 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44307
I20260925 22:49:50.700124 9253 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:50.700704 9253 heartbeater.cc:511] Master 127.4.219.190:44307 requested a full tablet report, sending...
I20260925 22:49:50.702344 9063 ts_manager.cc:194] Registered new tserver with Master: 2b684fa3b3034cb1aaaaa74900da3b7a (127.4.219.129:42511)
I20260925 22:49:50.703512 9063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:56849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:50.715517 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:50.827694 9258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:50.828020 9258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:50.828104 9258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:50.832154 9258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:50.832262 9258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:50.832361 9258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:50.837822 9258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:50.839044 9258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:50.840101 9258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:50.849412 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.849501 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:50.849416 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:50.849707 9258 server_base.cc:1034] running on GCE node
I20260925 22:49:50.850175 9258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:50.850831 9258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:50.851984 9258 hybrid_clock.cc:648] HybridClock initialized: now 1790376590851969 us; error 25 us; skew 500 ppm
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:50.854895 9258 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:50.855609 9258 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:50.855670 9258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:50.855845 9258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:50.857941 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "c075d21cf4bc42a482887a7159f30609"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "22bd1c5bb04b51a8ea1386048c30b296"
server_key_iv: "dec96561ca04d0fdad0a0859f7fb0cf4"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.858451 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "c075d21cf4bc42a482887a7159f30609"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "22bd1c5bb04b51a8ea1386048c30b296"
server_key_iv: "dec96561ca04d0fdad0a0859f7fb0cf4"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.862077 9258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:50.864390 9272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.865645 9258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:49:50.865761 9258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "c075d21cf4bc42a482887a7159f30609"
format_stamp: "Formatted at 2026-09-25 22:49:50 on dist-test-slave-kb1g"
server_key: "22bd1c5bb04b51a8ea1386048c30b296"
server_key_iv: "dec96561ca04d0fdad0a0859f7fb0cf4"
server_key_version: "encryptionkey@0"
I20260925 22:49:50.865846 9258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:50.895529 9258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:50.898716 9258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:50.898871 9258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:50.899416 9258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:50.900295 9258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:50.900343 9258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.900385 9258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:50.900400 9258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:50.909603 9258 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40405
I20260925 22:49:50.909651 9385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40405 every 8 connection(s)
I20260925 22:49:50.910329 9258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:50.912340 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9258
I20260925 22:49:50.912482 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:50.912787 4974 external_mini_cluster.cc:1428] Setting key 089736719a617b82c039ac2ea61a98bc
Sep 25 22:49:50 dist-test-slave-kb1g krb5kdc[9019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376590, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:49:50.926878 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.912314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:51113 (local address 127.4.219.190:44307)
0925 22:49:50.912485 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:50.912493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:50.913970 (+ 1477us) server_negotiation.cc:407] Connection header received
0925 22:49:50.915078 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:50.915081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:50.915124 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:50.915194 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:50.916903 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.917723 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.919597 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.919814 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.919892 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.923013 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:50.923016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:50.923018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:50.923044 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:50.924628 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.925219 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.925222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.925223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.925308 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:50.925720 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:50.925722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:50.925723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:50.925919 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:50.926009 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:50.926626 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 22:49:50.926740 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:49:50.928143 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:50.912852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44307 (local address 127.4.219.130:51113)
0925 22:49:50.913801 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:50.913864 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:50.914856 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:50.915391 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:50.915403 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:50.915959 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:50.916722 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.916749 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.917873 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.917880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.919457 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:50.919470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:50.919980 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:50.919987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:50.920242 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:50.920987 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:50.921010 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:50.922825 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:50.924798 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.924803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.924818 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.925107 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.925417 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:50.925420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:50.925422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:50.925640 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:50.926035 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:50.926045 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:50.926364 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 22:49:50.926558 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 22:49:50.926952 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":173,"threads_started":1}
I20260925 22:49:50.929517 9386 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44307
I20260925 22:49:50.929755 9386 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:50.930187 9386 heartbeater.cc:511] Master 127.4.219.190:44307 requested a full tablet report, sending...
I20260925 22:49:50.931339 9063 ts_manager.cc:194] Registered new tserver with Master: c075d21cf4bc42a482887a7159f30609 (127.4.219.130:40405)
I20260925 22:49:50.932327 9063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:51113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:50.943997 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:51.057809 9391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:51.058079 9391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:51.058128 9391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:51.062270 9391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:51.062332 9391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:51.062395 9391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:51.067693 9391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:51.068943 9391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:51.070014 9391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:51.079592 9399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.079684 9391 server_base.cc:1034] running on GCE node
W20260925 22:49:51.079579 9397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.079582 9396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.080305 9391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:51.081043 9391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:51.082204 9391 hybrid_clock.cc:648] HybridClock initialized: now 1790376591082188 us; error 30 us; skew 500 ppm
Sep 25 22:49:51 dist-test-slave-kb1g krb5kdc[9019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376591, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:51.085114 9391 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:51.085829 9391 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:51.085891 9391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:51.086058 9391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:51.088150 9391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "1025cc545cb2cc0ba40148e06137cbd4"
server_key_iv: "453e3a672acb56a7310c1834b0b8002c"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.088760 9391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "1025cc545cb2cc0ba40148e06137cbd4"
server_key_iv: "453e3a672acb56a7310c1834b0b8002c"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.092746 9391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:49:51.095224 9405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.096592 9391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:51.096725 9391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "1025cc545cb2cc0ba40148e06137cbd4"
server_key_iv: "453e3a672acb56a7310c1834b0b8002c"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.096843 9391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:51.107988 9391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:51.111224 9391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:51.111467 9391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:51.112068 9391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:51.113103 9391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:51.113175 9391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.113255 9391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:51.113307 9391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.123001 9391 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:33921
I20260925 22:49:51.123018 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:33921 every 8 connection(s)
I20260925 22:49:51.123672 9391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:51.130944 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9391
I20260925 22:49:51.131089 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:51.131388 4974 external_mini_cluster.cc:1428] Setting key 3a0fe67e7698e6218e2b62ca4b1de1fe
Sep 25 22:49:51 dist-test-slave-kb1g krb5kdc[9019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376591, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:51.138077 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.125587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:42499 (local address 127.4.219.190:44307)
0925 22:49:51.125735 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.125739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.126621 (+ 882us) server_negotiation.cc:407] Connection header received
0925 22:49:51.127946 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.127948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.127996 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.128075 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.129354 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.129936 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.131090 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.131415 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.131525 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.134499 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.134501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.134503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:51.134527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.136114 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:51.136630 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:51.136632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:51.136633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:51.136706 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:51.137005 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:51.137007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:51.137008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:51.137177 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.137283 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.137852 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.137958 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:51.138976 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.126004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44307 (local address 127.4.219.131:42499)
0925 22:49:51.126460 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.126504 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.127730 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.128248 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.128255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.128709 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:51.129221 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.129235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.130046 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.130049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.130937 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.130946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.131709 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.131716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.131878 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.132642 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:51.132659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:51.134347 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:51.136206 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:51.136211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:51.136224 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:51.136486 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:51.136784 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:51.136786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:51.136787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:51.136921 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:51.137306 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:51.137311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:51.137623 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.137785 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.138051 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260925 22:49:51.140311 9519 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44307
I20260925 22:49:51.140605 9519 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:51.141175 9519 heartbeater.cc:511] Master 127.4.219.190:44307 requested a full tablet report, sending...
I20260925 22:49:51.142151 9063 ts_manager.cc:194] Registered new tserver with Master: 9acb4f2c15af42e39c9652f0c5ab8b9c (127.4.219.131:33921)
I20260925 22:49:51.142729 9063 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:42499
I20260925 22:49:51.145560 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:51.155483 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.147473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.4.219.190:44307)
0925 22:49:51.147609 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.147614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.147841 (+ 227us) server_negotiation.cc:407] Connection header received
0925 22:49:51.147937 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.147939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.147977 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.148040 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.148801 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.149320 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.150072 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.150282 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.150356 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.151771 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.151774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.151776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:51.151798 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.153410 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:51.154019 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:51.154024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:51.154025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:51.154112 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:51.154556 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:51.154559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:51.154560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:51.154785 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.154891 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.155122 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.155250 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:51.158486 9063 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:51.161702 9063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:51.177191 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.170908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33921 (local address 127.0.0.1:45352)
0925 22:49:51.171680 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.171737 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.172240 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.172870 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.172877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.172900 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.173066 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.173073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174194 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.174199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.175492 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.175500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.176593 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.176597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.176711 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.176766 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.176892 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.177025 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":91,"threads_started":1}
I20260925 22:49:51.177220 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.171741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.4.219.131:33921)
0925 22:49:51.172197 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.172207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.172233 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:49:51.172428 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.172434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.172700 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.172865 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.173202 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174048 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.175732 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.176440 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.176587 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.176744 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.176937 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.177040 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":152,"threads_started":1}
I20260925 22:49:51.177593 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.170918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40405 (local address 127.0.0.1:54568)
0925 22:49:51.172113 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.172128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.172249 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.173066 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.173069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.173086 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.173189 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.173195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174671 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.174675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.175980 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.175990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.177144 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.177147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.177251 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.177276 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.177367 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.177442 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":87,"thread_start_us":104,"threads_started":1}
I20260925 22:49:51.177932 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.171891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.4.219.129:42511)
0925 22:49:51.172500 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.172505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.172524 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:51.172672 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.172677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.172857 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.173003 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.173370 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174509 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.176188 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.177024 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.177130 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.177264 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.177690 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.177791 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":67,"threads_started":1}
I20260925 22:49:51.178120 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.170992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.4.219.130:40405)
0925 22:49:51.171977 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.171989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.172677 (+ 688us) server_negotiation.cc:407] Connection header received
0925 22:49:51.172756 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.172765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.172948 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.173090 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.173301 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174509 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.176121 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.176863 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.176955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.177086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.177829 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.177938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":90,"threads_started":1}
I20260925 22:49:51.179062 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.171139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:42511 (local address 127.0.0.1:38714)
0925 22:49:51.172461 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.172473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.172576 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.173155 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.173161 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.173192 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.173271 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.173276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.174671 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.174675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.176063 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.176072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.177339 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.177344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.177435 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.177456 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.177523 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.177599 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1269,"mutex_wait_us":185,"thread_start_us":87,"threads_started":1}
I20260925 22:49:51.180508 9453 tablet_service.cc:1467] Processing CreateTablet for tablet 81b780f410a24ec1b6b808710955bf7a (DEFAULT_TABLE table=test-table [id=a8872f08d2ea4241ad698cdd81de0556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.180794 9320 tablet_service.cc:1467] Processing CreateTablet for tablet 81b780f410a24ec1b6b808710955bf7a (DEFAULT_TABLE table=test-table [id=a8872f08d2ea4241ad698cdd81de0556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.181649 9453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b780f410a24ec1b6b808710955bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.181890 9320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b780f410a24ec1b6b808710955bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.182585 9187 tablet_service.cc:1467] Processing CreateTablet for tablet 81b780f410a24ec1b6b808710955bf7a (DEFAULT_TABLE table=test-table [id=a8872f08d2ea4241ad698cdd81de0556]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:51.183593 9187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b780f410a24ec1b6b808710955bf7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.187628 9536 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Bootstrap starting.
I20260925 22:49:51.189890 9536 tablet_bootstrap.cc:654] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.190011 9537 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Bootstrap starting.
I20260925 22:49:51.191134 9536 log.cc:824] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.191766 9538 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Bootstrap starting.
I20260925 22:49:51.192121 9537 tablet_bootstrap.cc:654] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.193389 9536 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: No bootstrap required, opened a new log
I20260925 22:49:51.193391 9537 log.cc:824] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.193588 9536 ts_tablet_manager.cc:1397] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:49:51.193933 9538 tablet_bootstrap.cc:654] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.195186 9537 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: No bootstrap required, opened a new log
I20260925 22:49:51.195184 9538 log.cc:824] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.195405 9537 ts_tablet_manager.cc:1397] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:49:51.197299 9538 tablet_bootstrap.cc:492] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: No bootstrap required, opened a new log
I20260925 22:49:51.197484 9538 ts_tablet_manager.cc:1397] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 22:49:51.197707 9536 raft_consensus.cc:348] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.198011 9536 raft_consensus.cc:374] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.198063 9536 raft_consensus.cc:729] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acb4f2c15af42e39c9652f0c5ab8b9c, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.198568 9536 consensus_queue.cc:260] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.198875 9537 raft_consensus.cc:348] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.199164 9537 raft_consensus.cc:374] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.199234 9537 raft_consensus.cc:729] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c075d21cf4bc42a482887a7159f30609, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.199726 9537 consensus_queue.cc:260] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.199792 9536 ts_tablet_manager.cc:1428] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:49:51.200151 9519 heartbeater.cc:503] Master 127.4.219.190:44307 was elected leader, sending a full tablet report...
I20260925 22:49:51.200394 9386 heartbeater.cc:503] Master 127.4.219.190:44307 was elected leader, sending a full tablet report...
I20260925 22:49:51.200745 9537 ts_tablet_manager.cc:1428] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:49:51.201148 9538 raft_consensus.cc:348] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.201402 9538 raft_consensus.cc:374] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.201460 9538 raft_consensus.cc:729] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b684fa3b3034cb1aaaaa74900da3b7a, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.201889 9538 consensus_queue.cc:260] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.202495 9253 heartbeater.cc:503] Master 127.4.219.190:44307 was elected leader, sending a full tablet report...
I20260925 22:49:51.202821 9538 ts_tablet_manager.cc:1428] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:49:51.374643 9520 tablet.cc:2369] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:51.411639 9387 tablet.cc:2369] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:51.416810 9544 raft_consensus.cc:482] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:51.417018 9544 raft_consensus.cc:504] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.418232 9544 leader_election.cc:288] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c075d21cf4bc42a482887a7159f30609 (127.4.219.130:40405), 9acb4f2c15af42e39c9652f0c5ab8b9c (127.4.219.131:33921)
I20260925 22:49:51.424064 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.418625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43007 (local address 127.4.219.131:33921)
0925 22:49:51.418764 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.418768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.419091 (+ 323us) server_negotiation.cc:407] Connection header received
0925 22:49:51.419162 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.419165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.419218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.419319 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.419814 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.420629 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.422615 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.423269 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.423412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.423464 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.423846 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.423928 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:51.424064 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.418589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:40355 (local address 127.4.219.130:40405)
0925 22:49:51.418732 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.418739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.419089 (+ 350us) server_negotiation.cc:407] Connection header received
0925 22:49:51.419162 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.419165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.419218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.419315 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:51.419814 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.420653 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.422615 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.423269 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.423412 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.423464 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.423865 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.423936 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:51.424139 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.418541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33921 (local address 127.4.219.129:43007)
0925 22:49:51.418966 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.418979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.419067 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.419341 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.419344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.419381 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.419664 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.419671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.420756 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.420759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.422454 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.422466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.423429 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.423433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.423625 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.423731 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.423801 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.423956 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":290,"spinlock_wait_cycles":33408,"thread_start_us":111,"threads_started":1}
I20260925 22:49:51.424145 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.418478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40405 (local address 127.4.219.129:40355)
0925 22:49:51.418913 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.418929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.419061 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.419341 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.419344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.419381 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:51.419664 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.419671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.420756 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.420759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.422454 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:51.422466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.423429 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.423433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:51.423627 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.423717 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.423801 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.423956 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":417,"spinlock_wait_cycles":5760,"thread_start_us":130,"threads_started":1}
I20260925 22:49:51.424932 9473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b780f410a24ec1b6b808710955bf7a" candidate_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" is_pre_election: true
I20260925 22:49:51.424942 9340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b780f410a24ec1b6b808710955bf7a" candidate_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c075d21cf4bc42a482887a7159f30609" is_pre_election: true
I20260925 22:49:51.425256 9473 raft_consensus.cc:2454] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b684fa3b3034cb1aaaaa74900da3b7a in term 0.
I20260925 22:49:51.425266 9340 raft_consensus.cc:2454] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b684fa3b3034cb1aaaaa74900da3b7a in term 0.
I20260925 22:49:51.425786 9142 leader_election.cc:302] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b684fa3b3034cb1aaaaa74900da3b7a, 9acb4f2c15af42e39c9652f0c5ab8b9c, c075d21cf4bc42a482887a7159f30609; no voters:
I20260925 22:49:51.426097 9544 raft_consensus.cc:2781] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:51.426186 9544 raft_consensus.cc:482] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:51.426218 9544 raft_consensus.cc:3037] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.427274 9544 raft_consensus.cc:504] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
I20260925 22:49:51.427673 9544 leader_election.cc:288] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [CANDIDATE]: Term 1 election: Requested vote from peers c075d21cf4bc42a482887a7159f30609 (127.4.219.130:40405), 9acb4f2c15af42e39c9652f0c5ab8b9c (127.4.219.131:33921)
I20260925 22:49:51.428030 9340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b780f410a24ec1b6b808710955bf7a" candidate_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c075d21cf4bc42a482887a7159f30609"
I20260925 22:49:51.428140 9473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b780f410a24ec1b6b808710955bf7a" candidate_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c"
I20260925 22:49:51.428148 9340 raft_consensus.cc:3037] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.428294 9473 raft_consensus.cc:3037] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.429278 9340 raft_consensus.cc:2454] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b684fa3b3034cb1aaaaa74900da3b7a in term 1.
I20260925 22:49:51.429502 9473 raft_consensus.cc:2454] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b684fa3b3034cb1aaaaa74900da3b7a in term 1.
I20260925 22:49:51.429605 9141 leader_election.cc:302] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b684fa3b3034cb1aaaaa74900da3b7a, c075d21cf4bc42a482887a7159f30609; no voters:
I20260925 22:49:51.429796 9544 raft_consensus.cc:2781] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:51.430006 9544 raft_consensus.cc:686] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [term 1 LEADER]: Becoming Leader. State: Replica: 2b684fa3b3034cb1aaaaa74900da3b7a, State: Running, Role: LEADER
I20260925 22:49:51.430343 9544 consensus_queue.cc:237] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } }
W20260925 22:49:51.432173 9254 tablet.cc:2369] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:51.433670 9062 catalog_manager.cc:5495] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b684fa3b3034cb1aaaaa74900da3b7a (127.4.219.129). New cstate: current_term: 1 leader_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b684fa3b3034cb1aaaaa74900da3b7a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42511 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:51.481382 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.478136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.4.219.129:42511)
0925 22:49:51.478281 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.478285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.478297 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:51.478372 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.478375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.478428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.478518 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:51.478749 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.479242 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.480048 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:51.480251 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:51.480323 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:51.480633 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:51.481066 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:51.481159 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.481227 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.481274 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:49:51.487174 9473 raft_consensus.cc:1261] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 1 FOLLOWER]: Refusing update from remote peer 2b684fa3b3034cb1aaaaa74900da3b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:51.487174 9340 raft_consensus.cc:1261] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 1 FOLLOWER]: Refusing update from remote peer 2b684fa3b3034cb1aaaaa74900da3b7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:51.487919 9547 consensus_queue.cc:1035] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9acb4f2c15af42e39c9652f0c5ab8b9c" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:51.488094 9544 consensus_queue.cc:1035] T 81b780f410a24ec1b6b808710955bf7a P 2b684fa3b3034cb1aaaaa74900da3b7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c075d21cf4bc42a482887a7159f30609" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:51.494345 9550 mvcc.cc:205] Tried to move back new op lower bound from 7333382518724210688 to 7333382518501347328. Current Snapshot: MvccSnapshot[applied={T|T < 7333382518724210688}]
I20260925 22:49:51.495443 9552 mvcc.cc:205] Tried to move back new op lower bound from 7333382518724210688 to 7333382518501347328. Current Snapshot: MvccSnapshot[applied={T|T < 7333382518724210688}]
I20260925 22:49:51.496299 9551 mvcc.cc:205] Tried to move back new op lower bound from 7333382518724210688 to 7333382518501347328. Current Snapshot: MvccSnapshot[applied={T|T < 7333382518724210688}]
I20260925 22:49:51.504091 9062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:51.504287 9062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:51.509917 9062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b016f9f1a84d49d28c44f34217c0aad8: Sending DeleteTablet for 3 replicas of tablet 81b780f410a24ec1b6b808710955bf7a
I20260925 22:49:51.510902 9453 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b780f410a24ec1b6b808710955bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:45352
I20260925 22:49:51.511479 9559 tablet_replica.cc:330] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: stopping tablet replica
I20260925 22:49:51.511484 9187 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b780f410a24ec1b6b808710955bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:38714
I20260925 22:49:51.511857 9559 raft_consensus.cc:2229] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:51.511976 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9125
I20260925 22:49:51.512768 9559 raft_consensus.cc:2258] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:51.512837 9320 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b780f410a24ec1b6b808710955bf7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:51 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:54568
I20260925 22:49:51.513805 9561 tablet_replica.cc:330] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: stopping tablet replica
I20260925 22:49:51.514150 9561 raft_consensus.cc:2229] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:51.514433 9561 raft_consensus.cc:2258] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:51.514542 9559 ts_tablet_manager.cc:1905] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:51.515836 9561 ts_tablet_manager.cc:1905] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:51.517343 9559 ts_tablet_manager.cc:1918] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:51.517438 9559 log.cc:1196] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/81b780f410a24ec1b6b808710955bf7a
I20260925 22:49:51.517763 9559 ts_tablet_manager.cc:1939] T 81b780f410a24ec1b6b808710955bf7a P 9acb4f2c15af42e39c9652f0c5ab8b9c: Deleting consensus metadata
I20260925 22:49:51.518522 9561 ts_tablet_manager.cc:1918] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:51.518615 9561 log.cc:1196] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/81b780f410a24ec1b6b808710955bf7a
I20260925 22:49:51.518914 9561 ts_tablet_manager.cc:1939] T 81b780f410a24ec1b6b808710955bf7a P c075d21cf4bc42a482887a7159f30609: Deleting consensus metadata
I20260925 22:49:51.519132 9049 catalog_manager.cc:4841] TS 9acb4f2c15af42e39c9652f0c5ab8b9c (127.4.219.131:33921): tablet 81b780f410a24ec1b6b808710955bf7a (table test-table [id=a8872f08d2ea4241ad698cdd81de0556]) successfully deleted
I20260925 22:49:51.520190 9048 catalog_manager.cc:4841] TS c075d21cf4bc42a482887a7159f30609 (127.4.219.130:40405): tablet 81b780f410a24ec1b6b808710955bf7a (table test-table [id=a8872f08d2ea4241ad698cdd81de0556]) successfully deleted
W20260925 22:49:51.520277 9047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:42511 (error 108)
I20260925 22:49:51.521309 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.520995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:42511 (local address 127.0.0.1:38738)
0925 22:49:51.521120 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.521196 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:49:51.521389 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9258
W20260925 22:49:51.525645 9047 catalog_manager.cc:4584] TS 2b684fa3b3034cb1aaaaa74900da3b7a (127.4.219.129:42511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81b780f410a24ec1b6b808710955bf7a: Network error: Client connection negotiation failed: client connection to 127.4.219.129:42511: connect: Connection refused (error 111)
I20260925 22:49:51.530629 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9391
I20260925 22:49:51.537441 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9032
2026-09-25T22:49:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2528 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:51Z Disabled control of system clock
I20260925 22:49:51.571938 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37225
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:37225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:49:51.683719 9566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:51.684000 9566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:51.684036 9566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:51.688138 9566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:51.688205 9566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:51.688220 9566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:51.688236 9566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:51.688252 9566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:51.693075 9566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:37225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:51.694214 9566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:51.695641 9566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:51.704289 9571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.704284 9572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.704284 9574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.704355 9566 server_base.cc:1034] running on GCE node
I20260925 22:49:51.705102 9566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:51.706209 9566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:51.707377 9566 hybrid_clock.cc:648] HybridClock initialized: now 1790376591707358 us; error 30 us; skew 500 ppm
I20260925 22:49:51.709353 9566 webserver.cc:460] Webserver started at http://127.4.219.190:32859/ using document root <none> and password file <none>
I20260925 22:49:51.709898 9566 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:51.709957 9566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:51.710119 9566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:51.712217 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "84956c419d124dcc8a440f6dd114ede7"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "40218fc644aebfaed29c40b948473cea"
server_key_iv: "21afffd8ea43e09014334de1e84eb7ea"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.712823 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "84956c419d124dcc8a440f6dd114ede7"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "40218fc644aebfaed29c40b948473cea"
server_key_iv: "21afffd8ea43e09014334de1e84eb7ea"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.716447 9566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:49:51.719074 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.720592 9566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:51.720726 9566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "84956c419d124dcc8a440f6dd114ede7"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "40218fc644aebfaed29c40b948473cea"
server_key_iv: "21afffd8ea43e09014334de1e84eb7ea"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.720832 9566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:51.725790 9566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:51.726383 9566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:51.726531 9566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:51.734225 9566 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:37225
I20260925 22:49:51.734229 9632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:37225 every 8 connection(s)
I20260925 22:49:51.735299 9566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:51.737552 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9566
I20260925 22:49:51.737653 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:51.737965 4974 external_mini_cluster.cc:1428] Setting key 6a0ba5ec6e849584f8b66a93626d16c0
I20260925 22:49:51.738915 9633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:51.744423 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.739535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.4.219.190:37225)
0925 22:49:51.740244 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.740269 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.740315 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:49:51.741175 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.741200 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.741675 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.742068 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.742895 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.742901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.742917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:51.742928 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.742998 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:51.743021 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.743183 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.743260 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.743507 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260925 22:49:51.746205 9633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: Bootstrap starting.
I20260925 22:49:51.748591 9633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:51.749352 9633 log.cc:824] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:51.751276 9633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: No bootstrap required, opened a new log
I20260925 22:49:51.754788 9633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } }
I20260925 22:49:51.755023 9633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:51.755059 9633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84956c419d124dcc8a440f6dd114ede7, State: Initialized, Role: FOLLOWER
I20260925 22:49:51.755548 9633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } }
I20260925 22:49:51.755739 9633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:51.755810 9633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:51.755913 9633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:51.756909 9633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } }
I20260925 22:49:51.757251 9633 leader_election.cc:302] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84956c419d124dcc8a440f6dd114ede7; no voters:
I20260925 22:49:51.757555 9633 leader_election.cc:288] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:51.757714 9638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:51.757970 9638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [term 1 LEADER]: Becoming Leader. State: Replica: 84956c419d124dcc8a440f6dd114ede7, State: Running, Role: LEADER
I20260925 22:49:51.758258 9638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } }
I20260925 22:49:51.758792 9633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:51.759882 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84956c419d124dcc8a440f6dd114ede7. Latest consensus state: current_term: 1 leader_uuid: "84956c419d124dcc8a440f6dd114ede7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } } }
I20260925 22:49:51.760000 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:51.760370 9647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:51.760365 9639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84956c419d124dcc8a440f6dd114ede7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84956c419d124dcc8a440f6dd114ede7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37225 } } }
I20260925 22:49:51.760629 9639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:51.763621 9647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:51.769021 9647 catalog_manager.cc:1347] Generated new cluster ID: 694fb69defd347dd95112c9e7e8e89e1
I20260925 22:49:51.769086 9647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:51.784299 9647 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:51.784693 9647 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:51.793901 9647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: Generated new TSK 0
I20260925 22:49:51.794782 9647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:51.809232 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--builtin_ntp_servers=127.4.219.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:49:51.924631 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:51.924955 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:51.925030 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:51.929101 9657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:51.929159 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:51.929256 9657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:51.934006 9657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:51.935230 9657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:51.936995 9657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:51.946861 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.946847 9657 server_base.cc:1034] running on GCE node
W20260925 22:49:51.946803 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:51.946798 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:51.947569 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:51.948222 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:51.949424 9657 hybrid_clock.cc:648] HybridClock initialized: now 1790376591949386 us; error 51 us; skew 500 ppm
I20260925 22:49:51.951294 9657 webserver.cc:460] Webserver started at http://127.4.219.129:43213/ using document root <none> and password file <none>
I20260925 22:49:51.951880 9657 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:51.951972 9657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:51.952207 9657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:51.954347 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "0e97d2d8fe8a4f3d91d68f48216118a9"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "33d6ed7a73a57e4d373f4a74107092da"
server_key_iv: "8c5636368a0bddc2ac3ddc15906fdec0"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.954918 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "0e97d2d8fe8a4f3d91d68f48216118a9"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "33d6ed7a73a57e4d373f4a74107092da"
server_key_iv: "8c5636368a0bddc2ac3ddc15906fdec0"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.958758 9657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:51.961452 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.962846 9657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:51.962954 9657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "0e97d2d8fe8a4f3d91d68f48216118a9"
format_stamp: "Formatted at 2026-09-25 22:49:51 on dist-test-slave-kb1g"
server_key: "33d6ed7a73a57e4d373f4a74107092da"
server_key_iv: "8c5636368a0bddc2ac3ddc15906fdec0"
server_key_version: "encryptionkey@0"
I20260925 22:49:51.963136 9657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:51.968434 9657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:51.969178 9657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:51.969342 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:51.969884 9657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:51.970911 9657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:51.970963 9657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.971004 9657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:51.971021 9657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:51.981127 9657 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:34811
I20260925 22:49:51.981146 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:34811 every 8 connection(s)
I20260925 22:49:51.982155 9657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:51.985859 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9657
I20260925 22:49:51.985960 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:51.986263 4974 external_mini_cluster.cc:1428] Setting key 19fcc750598f54671d15605e3a5ab8f0
I20260925 22:49:51.988519 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--builtin_ntp_servers=127.4.219.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:49:51.988997 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.984190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:46819 (local address 127.4.219.190:37225)
0925 22:49:51.984339 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:51.984343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:51.985610 (+ 1267us) server_negotiation.cc:407] Connection header received
0925 22:49:51.987022 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:51.987029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:51.987084 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:51.987175 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:51.988502 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:51.988506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:51.988508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:51.988520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:51.988558 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:51.988568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:51.988681 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:51.988824 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 22:49:51.988867 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:49:51.990041 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:51.984611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37225 (local address 127.4.219.129:46819)
0925 22:49:51.985330 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:51.985372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:51.986806 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:51.987366 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:51.987375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:51.987490 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:51.988229 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:51.988231 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:51.988271 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:51.988341 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:51.988669 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:51.988677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:51.988683 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:49:51.988870 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:49:51.989127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260925 22:49:51.991408 9785 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37225
I20260925 22:49:51.991679 9785 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:51.992120 9785 heartbeater.cc:511] Master 127.4.219.190:37225 requested a full tablet report, sending...
I20260925 22:49:51.993983 9597 ts_manager.cc:194] Registered new tserver with Master: 0e97d2d8fe8a4f3d91d68f48216118a9 (127.4.219.129:34811)
W20260925 22:49:52.103291 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:52.103585 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:52.103626 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:52.107810 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:52.107882 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:52.107940 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:52.112931 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:52.114117 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:52.115634 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:52.125923 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.125923 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.125926 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:52.125964 9788 server_base.cc:1034] running on GCE node
I20260925 22:49:52.126636 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:52.127377 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:52.128602 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790376592128554 us; error 89 us; skew 500 ppm
I20260925 22:49:52.130515 9788 webserver.cc:460] Webserver started at http://127.4.219.130:37455/ using document root <none> and password file <none>
I20260925 22:49:52.131120 9788 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:52.131219 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:52.131490 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:52.133703 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "ea6194f49a874f87b0ea4761e103d3d0"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "1d296bf2f3a2a3427b4362ff64e54a48"
server_key_iv: "7de36ab70be54799be497a03122b6d8a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.134338 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "ea6194f49a874f87b0ea4761e103d3d0"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "1d296bf2f3a2a3427b4362ff64e54a48"
server_key_iv: "7de36ab70be54799be497a03122b6d8a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.138352 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:49:52.140969 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.142310 9788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:49:52.142478 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "ea6194f49a874f87b0ea4761e103d3d0"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "1d296bf2f3a2a3427b4362ff64e54a48"
server_key_iv: "7de36ab70be54799be497a03122b6d8a"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.142603 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:52.147547 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:52.148247 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:52.148588 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:52.149293 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:52.150302 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:52.150379 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.150481 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:52.150540 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.160512 9788 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:43377
I20260925 22:49:52.160573 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:43377 every 8 connection(s)
I20260925 22:49:52.161656 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:52.165480 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9788
I20260925 22:49:52.165630 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:52.165941 4974 external_mini_cluster.cc:1428] Setting key 370341d8d9888968516948d54ecf6062
I20260925 22:49:52.168112 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--builtin_ntp_servers=127.4.219.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:49:52.168601 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.163693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:42431 (local address 127.4.219.190:37225)
0925 22:49:52.163851 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.163856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.165107 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 22:49:52.166614 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.166618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.166682 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.166777 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.168078 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.168082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.168085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.168096 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.168117 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.168128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.168255 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.168398 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.168443 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:49:52.169477 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.164226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37225 (local address 127.4.219.130:42431)
0925 22:49:52.164944 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.164986 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.166373 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.166957 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.166967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.167084 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.167820 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.167822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.167866 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.167936 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.168242 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.168250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.168255 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.168444 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.168725 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260925 22:49:52.170833 9916 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37225
I20260925 22:49:52.171113 9916 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:52.171602 9916 heartbeater.cc:511] Master 127.4.219.190:37225 requested a full tablet report, sending...
I20260925 22:49:52.172842 9597 ts_manager.cc:194] Registered new tserver with Master: ea6194f49a874f87b0ea4761e103d3d0 (127.4.219.130:43377)
W20260925 22:49:52.295758 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:52.296155 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:52.296244 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:52.300997 9919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:52.301088 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:52.301189 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:52.306613 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:52.307809 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:52.309391 9919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:52.319386 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.319355 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.319367 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:52.319705 9919 server_base.cc:1034] running on GCE node
I20260925 22:49:52.320190 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:52.320976 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:52.322170 9919 hybrid_clock.cc:648] HybridClock initialized: now 1790376592322133 us; error 49 us; skew 500 ppm
I20260925 22:49:52.324335 9919 webserver.cc:460] Webserver started at http://127.4.219.131:40611/ using document root <none> and password file <none>
I20260925 22:49:52.325110 9919 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:52.325210 9919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:52.325465 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:52.327694 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "5460e1809dbc48238d4069f3abec0e21"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "ff5e515ac6c71dacc2fb9973c98078d0"
server_key_iv: "03e9ffa019ba8c8a105413d7669841d4"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.328295 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "5460e1809dbc48238d4069f3abec0e21"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "ff5e515ac6c71dacc2fb9973c98078d0"
server_key_iv: "03e9ffa019ba8c8a105413d7669841d4"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.332325 9919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:49:52.335083 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.336418 9919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:52.336606 9919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "5460e1809dbc48238d4069f3abec0e21"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "ff5e515ac6c71dacc2fb9973c98078d0"
server_key_iv: "03e9ffa019ba8c8a105413d7669841d4"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.336766 9919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:52.344126 9919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:52.344887 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:52.345042 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:52.345587 9919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:52.346639 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:52.346697 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.346738 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:52.346791 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.355990 9919 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:38199
I20260925 22:49:52.356011 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:38199 every 8 connection(s)
I20260925 22:49:52.357089 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:52.364195 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.359262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44033 (local address 127.4.219.190:37225)
0925 22:49:52.359908 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.359914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.360706 (+ 792us) server_negotiation.cc:407] Connection header received
0925 22:49:52.362243 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.362247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.362294 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.362386 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.363675 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.363678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.363682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.363695 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.363718 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.363734 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.363835 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.363989 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.364037 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:52.365227 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.359662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37225 (local address 127.4.219.131:44033)
0925 22:49:52.360482 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.360581 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.361988 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.362594 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.362601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.362701 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.363381 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.363383 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.363428 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.363503 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.363825 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.363834 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.363840 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.364030 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.364310 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260925 22:49:52.366451 10047 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37225
I20260925 22:49:52.366765 10047 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:52.367295 10047 heartbeater.cc:511] Master 127.4.219.190:37225 requested a full tablet report, sending...
I20260925 22:49:52.367391 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 9919
I20260925 22:49:52.367509 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:52.367810 4974 external_mini_cluster.cc:1428] Setting key d5747b70eced3786e8d1b359e3aa52fa
I20260925 22:49:52.368736 9596 ts_manager.cc:194] Registered new tserver with Master: 5460e1809dbc48238d4069f3abec0e21 (127.4.219.131:38199)
I20260925 22:49:52.369938 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:49:52.373796 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.372006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34896 (local address 127.4.219.190:37225)
0925 22:49:52.372167 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.372171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.372339 (+ 168us) server_negotiation.cc:407] Connection header received
0925 22:49:52.372497 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.372501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.372567 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.372640 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.373359 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.373364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.373366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.373377 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.373398 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.373407 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.373506 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.373577 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.373642 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:52.377780 9596 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:49:52.381498 9596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:49:52.394470 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:38199 (local address 127.0.0.1:43526)
0925 22:49:52.391973 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.391988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.392067 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.392876 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.392883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.392964 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.393694 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.393697 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.393719 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.393786 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.394113 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.394121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.394128 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.394231 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.394305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":133,"threads_started":1}
I20260925 22:49:52.394490 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.4.219.131:38199)
0925 22:49:52.392404 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.392416 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.392435 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:52.392477 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.392482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.392690 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.393267 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.393916 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.393923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.393935 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.393945 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.393986 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.394014 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.394132 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.394258 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.394341 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":189,"threads_started":1}
I20260925 22:49:52.395581 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34811 (local address 127.0.0.1:54686)
0925 22:49:52.392620 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.392635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.392713 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.393595 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.393598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.393612 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.394930 (+ 1318us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.394932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.394942 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.394968 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.395311 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.395314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.395319 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.395405 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.395441 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":81,"threads_started":1}
I20260925 22:49:52.395820 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43377 (local address 127.0.0.1:49718)
0925 22:49:52.391526 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.391576 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.391694 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.392879 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.392887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.392963 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.394596 (+ 1633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.394599 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.394620 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.394650 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.395624 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.395629 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.395634 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.395689 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.395722 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":107,"threads_started":1}
I20260925 22:49:52.396093 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.4.219.129:34811)
0925 22:49:52.392362 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.392374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.392802 (+ 428us) server_negotiation.cc:407] Connection header received
0925 22:49:52.392893 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.392898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.393074 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.393812 (+ 738us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.395116 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.395122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.395133 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.395144 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.395189 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.395219 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.395325 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.395579 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.395933 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":97,"threads_started":1}
I20260925 22:49:52.396471 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.391107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.4.219.130:43377)
0925 22:49:52.392362 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.392374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.392400 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:49:52.392460 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.392465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.392690 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.392875 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.395037 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.395047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.395058 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.395072 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.395465 (+ 393us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.395506 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.396173 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.396232 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.396298 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":92,"threads_started":1}
I20260925 22:49:52.397987 9981 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 (DEFAULT_TABLE table=test-table [id=492d45cfd39b4c01bc9d311a3050a151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:52.399158 9981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ee79a78ba42628ba09b2e4ce64125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:52.399026 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 (DEFAULT_TABLE table=test-table [id=492d45cfd39b4c01bc9d311a3050a151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:52.399225 9850 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 (DEFAULT_TABLE table=test-table [id=492d45cfd39b4c01bc9d311a3050a151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:49:52.400053 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ee79a78ba42628ba09b2e4ce64125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:52.400202 9850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0ee79a78ba42628ba09b2e4ce64125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:52.407083 10064 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Bootstrap starting.
I20260925 22:49:52.407553 10065 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Bootstrap starting.
I20260925 22:49:52.408361 10066 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Bootstrap starting.
I20260925 22:49:52.410326 10066 tablet_bootstrap.cc:654] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:52.410356 10064 tablet_bootstrap.cc:654] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:52.411101 10066 log.cc:824] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:52.411832 10065 tablet_bootstrap.cc:654] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:52.413170 10066 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: No bootstrap required, opened a new log
I20260925 22:49:52.413314 10065 log.cc:824] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:52.413331 10064 log.cc:824] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:52.413388 10066 ts_tablet_manager.cc:1397] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:49:52.415400 10065 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: No bootstrap required, opened a new log
I20260925 22:49:52.415397 10064 tablet_bootstrap.cc:492] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: No bootstrap required, opened a new log
I20260925 22:49:52.415591 10065 ts_tablet_manager.cc:1397] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260925 22:49:52.415591 10064 ts_tablet_manager.cc:1397] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 22:49:52.418397 10066 raft_consensus.cc:348] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.418730 10066 raft_consensus.cc:374] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:52.418788 10066 raft_consensus.cc:729] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e97d2d8fe8a4f3d91d68f48216118a9, State: Initialized, Role: FOLLOWER
I20260925 22:49:52.419427 10066 consensus_queue.cc:260] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.420395 10065 raft_consensus.cc:348] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.420348 10064 raft_consensus.cc:348] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.420591 10066 ts_tablet_manager.cc:1428] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:49:52.420648 10064 raft_consensus.cc:374] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:52.420648 10065 raft_consensus.cc:374] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:52.420707 10064 raft_consensus.cc:729] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460e1809dbc48238d4069f3abec0e21, State: Initialized, Role: FOLLOWER
I20260925 22:49:52.420718 10065 raft_consensus.cc:729] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6194f49a874f87b0ea4761e103d3d0, State: Initialized, Role: FOLLOWER
I20260925 22:49:52.420974 9785 heartbeater.cc:503] Master 127.4.219.190:37225 was elected leader, sending a full tablet report...
I20260925 22:49:52.421182 10064 consensus_queue.cc:260] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.421191 10065 consensus_queue.cc:260] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.422201 10064 ts_tablet_manager.cc:1428] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:49:52.422623 10047 heartbeater.cc:503] Master 127.4.219.190:37225 was elected leader, sending a full tablet report...
I20260925 22:49:52.424387 10065 ts_tablet_manager.cc:1428] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260925 22:49:52.424793 9916 heartbeater.cc:503] Master 127.4.219.190:37225 was elected leader, sending a full tablet report...
I20260925 22:49:52.453864 10070 raft_consensus.cc:482] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:49:52.454169 10070 raft_consensus.cc:504] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.455891 10070 leader_election.cc:288] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea6194f49a874f87b0ea4761e103d3d0 (127.4.219.130:43377), 5460e1809dbc48238d4069f3abec0e21 (127.4.219.131:38199)
I20260925 22:49:52.458006 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.456378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:49433 (local address 127.4.219.130:43377)
0925 22:49:52.456558 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.456565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.456587 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:49:52.456639 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.456643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.456674 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.456782 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.457563 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.457568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.457573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.457585 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.457609 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.457621 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.457732 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.457819 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.457861 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:49:52.458005 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.456566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:60169 (local address 127.4.219.131:38199)
0925 22:49:52.456697 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.456704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.456773 (+ 69us) server_negotiation.cc:407] Connection header received
0925 22:49:52.456904 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.456907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.456951 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.457059 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.457597 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.457600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.457603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.457615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.457646 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.457660 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.457753 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.457819 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.457861 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:52.458022 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.456185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43377 (local address 127.4.219.129:49433)
0925 22:49:52.456301 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.456317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.456408 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.456803 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.456806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.456821 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.457332 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.457334 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.457343 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.457392 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.457735 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.457738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.457741 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.457821 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.457861 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 22:49:52.458060 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.456241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:38199 (local address 127.4.219.129:60169)
0925 22:49:52.456688 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.456704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.456807 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.457089 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.457091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.457102 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:52.457443 (+ 341us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:49:52.457448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:52.457459 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:49:52.457483 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:52.457741 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:52.457743 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:52.457745 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:49:52.457821 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:49:52.457868 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":3,"thread_start_us":141,"threads_started":1}
I20260925 22:49:52.458751 9870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ee79a78ba42628ba09b2e4ce64125" candidate_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6194f49a874f87b0ea4761e103d3d0" is_pre_election: true
I20260925 22:49:52.458894 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ee79a78ba42628ba09b2e4ce64125" candidate_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460e1809dbc48238d4069f3abec0e21" is_pre_election: true
I20260925 22:49:52.459064 9870 raft_consensus.cc:2454] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e97d2d8fe8a4f3d91d68f48216118a9 in term 0.
I20260925 22:49:52.459199 10001 raft_consensus.cc:2454] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e97d2d8fe8a4f3d91d68f48216118a9 in term 0.
I20260925 22:49:52.459606 9672 leader_election.cc:302] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e97d2d8fe8a4f3d91d68f48216118a9, ea6194f49a874f87b0ea4761e103d3d0; no voters:
I20260925 22:49:52.459925 10070 raft_consensus.cc:2781] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:49:52.460007 10070 raft_consensus.cc:482] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:49:52.460078 10070 raft_consensus.cc:3037] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:49:52.483500 9786 tablet.cc:2369] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:52.488958 10070 raft_consensus.cc:504] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.489502 10070 leader_election.cc:288] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [CANDIDATE]: Term 1 election: Requested vote from peers ea6194f49a874f87b0ea4761e103d3d0 (127.4.219.130:43377), 5460e1809dbc48238d4069f3abec0e21 (127.4.219.131:38199)
I20260925 22:49:52.490044 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ee79a78ba42628ba09b2e4ce64125" candidate_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460e1809dbc48238d4069f3abec0e21"
I20260925 22:49:52.490033 9870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0ee79a78ba42628ba09b2e4ce64125" candidate_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea6194f49a874f87b0ea4761e103d3d0"
I20260925 22:49:52.490231 9870 raft_consensus.cc:3037] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:52.490221 10001 raft_consensus.cc:3037] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:52.491431 10001 raft_consensus.cc:2454] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e97d2d8fe8a4f3d91d68f48216118a9 in term 1.
I20260925 22:49:52.491886 9674 leader_election.cc:302] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e97d2d8fe8a4f3d91d68f48216118a9, 5460e1809dbc48238d4069f3abec0e21; no voters:
I20260925 22:49:52.491961 9870 raft_consensus.cc:2454] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e97d2d8fe8a4f3d91d68f48216118a9 in term 1.
I20260925 22:49:52.492123 10070 raft_consensus.cc:2781] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:52.492352 10070 raft_consensus.cc:686] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e97d2d8fe8a4f3d91d68f48216118a9, State: Running, Role: LEADER
I20260925 22:49:52.492784 10070 consensus_queue.cc:237] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } }
I20260925 22:49:52.496410 9597 catalog_manager.cc:5495] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e97d2d8fe8a4f3d91d68f48216118a9 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e97d2d8fe8a4f3d91d68f48216118a9" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34811 } health_report { overall_health: HEALTHY } } }
I20260925 22:49:52.530292 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.528775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.4.219.129:34811)
0925 22:49:52.528902 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.528906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.528919 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:49:52.529053 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.529056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.529119 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.529204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.529965 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.529968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.529971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.529983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.530001 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.530009 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.530084 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.530142 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.530178 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:49:52.537241 10001 raft_consensus.cc:1261] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 1 FOLLOWER]: Refusing update from remote peer 0e97d2d8fe8a4f3d91d68f48216118a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:52.537263 9870 raft_consensus.cc:1261] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 1 FOLLOWER]: Refusing update from remote peer 0e97d2d8fe8a4f3d91d68f48216118a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:52.538007 10074 consensus_queue.cc:1035] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea6194f49a874f87b0ea4761e103d3d0" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:52.538195 10070 consensus_queue.cc:1035] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5460e1809dbc48238d4069f3abec0e21" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:52.546591 10079 mvcc.cc:205] Tried to move back new op lower bound from 7333382523024912384 to 7333382522853654528. Current Snapshot: MvccSnapshot[applied={T|T < 7333382523024912384}]
I20260925 22:49:52.555395 9597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:52.555593 9597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:49:52.558032 9597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84956c419d124dcc8a440f6dd114ede7: Sending DeleteTablet for 3 replicas of tablet 7e0ee79a78ba42628ba09b2e4ce64125
I20260925 22:49:52.559005 9981 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:52 UTC) from {username='slave'} at 127.0.0.1:43526
I20260925 22:49:52.559055 9850 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:52 UTC) from {username='slave'} at 127.0.0.1:49718
I20260925 22:49:52.559077 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e0ee79a78ba42628ba09b2e4ce64125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:49:52 UTC) from {username='slave'} at 127.0.0.1:54686
I20260925 22:49:52.559567 10089 tablet_replica.cc:330] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9: stopping tablet replica
I20260925 22:49:52.559988 10089 raft_consensus.cc:2229] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:49:52.560171 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9657
I20260925 22:49:52.560439 10089 raft_consensus.cc:2258] T 7e0ee79a78ba42628ba09b2e4ce64125 P 0e97d2d8fe8a4f3d91d68f48216118a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:52.564824 10088 tablet_replica.cc:330] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: stopping tablet replica
I20260925 22:49:52.565208 10087 tablet_replica.cc:330] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: stopping tablet replica
I20260925 22:49:52.565245 10088 raft_consensus.cc:2229] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:52.565515 10087 raft_consensus.cc:2229] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:52.565621 10088 raft_consensus.cc:2258] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:52.565855 10087 raft_consensus.cc:2258] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:52.567248 10087 ts_tablet_manager.cc:1905] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:52.567318 10088 ts_tablet_manager.cc:1905] T 7e0ee79a78ba42628ba09b2e4ce64125 P ea6194f49a874f87b0ea4761e103d3d0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:49:52.568097 9584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:34811 (error 108)
I20260925 22:49:52.569175 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9788
I20260925 22:49:52.570079 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.569447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34811 (local address 127.0.0.1:54692)
0925 22:49:52.569885 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.569968 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
W20260925 22:49:52.570240 9584 catalog_manager.cc:4584] TS 0e97d2d8fe8a4f3d91d68f48216118a9 (127.4.219.129:34811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e0ee79a78ba42628ba09b2e4ce64125: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34811: connect: Connection refused (error 111)
I20260925 22:49:52.593115 10087 ts_tablet_manager.cc:1918] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:49:52.593283 10087 log.cc:1196] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/7e0ee79a78ba42628ba09b2e4ce64125
I20260925 22:49:52.593595 10087 ts_tablet_manager.cc:1939] T 7e0ee79a78ba42628ba09b2e4ce64125 P 5460e1809dbc48238d4069f3abec0e21: Deleting consensus metadata
I20260925 22:49:52.597378 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9919
I20260925 22:49:52.598820 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.598447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43377 (local address 127.0.0.1:49722)
0925 22:49:52.598624 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.598703 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.130:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:49:52.624296 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.623929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34811 (local address 127.0.0.1:54696)
0925 22:49:52.624118 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.624208 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:49:52.625175 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.624948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34811 (local address 127.0.0.1:54700)
0925 22:49:52.625062 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.625111 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:49:52.629354 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 9566
2026-09-25T22:49:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1087 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:49:52.651479 4974 mini_oidc.cc:152] Starting JWKS server
I20260925 22:49:52.653306 4974 webserver.cc:460] Webserver started at https://127.0.0.1:34951/ using document root <none> and password file <none>
I20260925 22:49:52.653505 4974 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:49:52.653810 4974 webserver.cc:460] Webserver started at http://127.0.0.1:38023/ using document root <none> and password file <none>
2026-09-25T22:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:52Z Disabled control of system clock
I20260925 22:49:52.664316 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35955
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:35955
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/testchainca.cert with env {}
W20260925 22:49:52.787448 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:52.787771 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:52.787814 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:52.792160 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:52.792235 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:52.792251 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:52.792269 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:52.792285 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:49:52.792307 10096 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:49:52.792320 10096 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34951/jwks.json
I20260925 22:49:52.798772 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:35955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:52.800239 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:52.801635 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:52.811019 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.811026 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:52.811086 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:52.811427 10096 server_base.cc:1034] running on GCE node
I20260925 22:49:52.811980 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:52.813201 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:52.814429 10096 hybrid_clock.cc:648] HybridClock initialized: now 1790376592814401 us; error 40 us; skew 500 ppm
I20260925 22:49:52.816679 10096 webserver.cc:460] Webserver started at http://127.4.219.190:45773/ using document root <none> and password file <none>
I20260925 22:49:52.817322 10096 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:52.817420 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:52.817694 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:52.819936 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "5e80e908a7f5459eba792dd2294fcfb3"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "8bf781ddace4bd5e6d1856ebdce32c18"
server_key_iv: "b07c761b00a684c572d51c89c0325abb"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.820623 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "5e80e908a7f5459eba792dd2294fcfb3"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "8bf781ddace4bd5e6d1856ebdce32c18"
server_key_iv: "b07c761b00a684c572d51c89c0325abb"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.824762 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:49:52.827780 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:52.829231 10096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:52.829381 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "5e80e908a7f5459eba792dd2294fcfb3"
format_stamp: "Formatted at 2026-09-25 22:49:52 on dist-test-slave-kb1g"
server_key: "8bf781ddace4bd5e6d1856ebdce32c18"
server_key_iv: "b07c761b00a684c572d51c89c0325abb"
server_key_version: "encryptionkey@0"
I20260925 22:49:52.829505 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:52.841464 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:52.842263 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:52.842480 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:52.850001 10096 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:35955
I20260925 22:49:52.850015 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:35955 every 8 connection(s)
I20260925 22:49:52.851116 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:52.854637 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:52.860251 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: Bootstrap starting.
I20260925 22:49:52.860870 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10096
I20260925 22:49:52.861002 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:52.861300 4974 external_mini_cluster.cc:1428] Setting key a1ddabf786ce977447327cc1f6c90632
I20260925 22:49:52.863281 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:52.864190 10163 log.cc:824] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:52.866305 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: No bootstrap required, opened a new log
I20260925 22:49:52.869944 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } }
I20260925 22:49:52.870186 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:52.870247 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e80e908a7f5459eba792dd2294fcfb3, State: Initialized, Role: FOLLOWER
I20260925 22:49:52.870774 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } }
I20260925 22:49:52.870957 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:52.871025 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:52.871134 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:52.871917 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.862616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.4.219.190:35955)
0925 22:49:52.863794 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.863810 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.863848 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:49:52.864669 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.864696 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.865225 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.865598 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:52.866059 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.867124 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.868129 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.868739 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.868924 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:52.869860 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:52.869865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:52.869880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:52.870381 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:52.870450 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:52.870471 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:52.870589 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:52.870749 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:49:52.871080 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":157,"threads_started":1}
I20260925 22:49:52.874886 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:52.900955 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } }
I20260925 22:49:52.901461 10163 leader_election.cc:302] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e80e908a7f5459eba792dd2294fcfb3; no voters:
I20260925 22:49:52.901853 10163 leader_election.cc:288] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:52.902002 10168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:52.902261 10168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 5e80e908a7f5459eba792dd2294fcfb3, State: Running, Role: LEADER
I20260925 22:49:52.902644 10168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } }
I20260925 22:49:52.903388 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:52.904881 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e80e908a7f5459eba792dd2294fcfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } } }
I20260925 22:49:52.905043 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:52.905019 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e80e908a7f5459eba792dd2294fcfb3. Latest consensus state: current_term: 1 leader_uuid: "5e80e908a7f5459eba792dd2294fcfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e80e908a7f5459eba792dd2294fcfb3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35955 } } }
I20260925 22:49:52.905094 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3 [sys.catalog]: This master's current role is: LEADER
W20260925 22:49:52.908313 10183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:52.908440 10183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:52.908514 10180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:52.909294 10180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:52.915555 10180 catalog_manager.cc:1347] Generated new cluster ID: 620c9d6948ca4b46a03d010446f2e9e9
I20260925 22:49:52.915769 10180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:52.928896 10180 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:52.930222 10180 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:52.936030 10180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e80e908a7f5459eba792dd2294fcfb3: Generated new TSK 0
I20260925 22:49:52.936919 10180 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:49:52.972729 10193 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:49:52.972862 10193 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:49:52.973443 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:52.949364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.4.219.190:35955)
0925 22:49:52.949521 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:52.949527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:52.949778 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:49:52.949981 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:52.949984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:52.950032 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:52.950125 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:49:52.950430 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.951044 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.952296 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.952519 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.952731 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:52.953076 (+ 345us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:49:52.972987 (+ 19911us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:49:52.973296 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 22:49:52.973531 10166 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:49:52.974709 10192 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:52.949274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:35955 (local address 127.0.0.1:58458)
0925 22:49:52.949653 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:52.949671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:52.949816 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:52.950173 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:52.950176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:52.950184 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:49:52.950318 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:52.950325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.951168 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.951172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:52.952167 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:52.952178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:52.952705 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:52.952710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:52.952808 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:52.952865 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:49:52.973277 (+ 20412us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:49:52.973528 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.219.190:35955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.219.190:35955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:49:52.975538 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10096
2026-09-25T22:49:52Z chronyd exiting
I20260925 22:49:53.013396 4974 test_util.cc:175] -----------------------------------------------
I20260925 22:49:53.013489 4974 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376566921023-4974-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (363 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:49:53.014564 4974 mini_oidc.cc:152] Starting JWKS server
I20260925 22:49:53.015584 4974 webserver.cc:460] Webserver started at https://127.0.0.1:33721/ using document root <none> and password file <none>
I20260925 22:49:53.015722 4974 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:49:53.016036 4974 webserver.cc:460] Webserver started at http://127.0.0.1:39089/ using document root <none> and password file <none>
2026-09-25T22:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:53Z Disabled control of system clock
I20260925 22:49:53.025309 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44221
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44221
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33721/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/testchainca.cert with env {}
W20260925 22:49:53.137673 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:53.138024 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:53.138108 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:53.142338 10199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:53.142452 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:53.142481 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:53.142519 10199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:53.142575 10199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:49:53.142624 10199 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:49:53.142658 10199 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33721/jwks.json
I20260925 22:49:53.148417 10199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33721/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:53.149688 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:53.150785 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:53.159655 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:53.159655 10205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:53.159655 10207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.160066 10199 server_base.cc:1034] running on GCE node
I20260925 22:49:53.160708 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:53.161854 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:53.163017 10199 hybrid_clock.cc:648] HybridClock initialized: now 1790376593162968 us; error 65 us; skew 500 ppm
I20260925 22:49:53.165019 10199 webserver.cc:460] Webserver started at http://127.4.219.190:40871/ using document root <none> and password file <none>
I20260925 22:49:53.165652 10199 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:53.165716 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:53.165961 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:53.168074 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "5bd72b554da044868dde89586cea8fa0"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "96f55503abc340a66ec3af5234f6bfcd"
server_key_iv: "b182c242e73f89d27868f7435fd91046"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.168720 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "5bd72b554da044868dde89586cea8fa0"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "96f55503abc340a66ec3af5234f6bfcd"
server_key_iv: "b182c242e73f89d27868f7435fd91046"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.172683 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:53.175349 10213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:53.176957 10199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:53.177081 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "5bd72b554da044868dde89586cea8fa0"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "96f55503abc340a66ec3af5234f6bfcd"
server_key_iv: "b182c242e73f89d27868f7435fd91046"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.177209 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:53.192507 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:53.193363 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:53.193533 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:53.201968 10199 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44221
I20260925 22:49:53.201985 10265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44221 every 8 connection(s)
I20260925 22:49:53.203136 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:53.206593 10266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:53.211592 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10199
I20260925 22:49:53.211786 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:53.212074 4974 external_mini_cluster.cc:1428] Setting key bcdf7f2981e96a8c44e985781edc95e7
I20260925 22:49:53.212749 10266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0: Bootstrap starting.
I20260925 22:49:53.215370 10266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:53.216296 10266 log.cc:824] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:53.218341 10266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0: No bootstrap required, opened a new log
I20260925 22:49:53.222402 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:53.213652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.4.219.190:44221)
0925 22:49:53.214403 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:53.214418 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:53.214459 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:49:53.215162 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:53.215182 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:53.215609 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:53.216008 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:53.216670 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.217711 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.218762 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.219296 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.219485 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.220398 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:53.220404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:53.220418 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:53.220933 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:53.220993 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:53.221016 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:53.221150 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:53.221300 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:49:53.221589 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260925 22:49:53.222267 10266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } }
I20260925 22:49:53.222573 10266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:53.222635 10266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd72b554da044868dde89586cea8fa0, State: Initialized, Role: FOLLOWER
I20260925 22:49:53.223127 10266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } }
I20260925 22:49:53.223282 10266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:53.223335 10266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:53.223417 10266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:53.225528 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:53.298524 10266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } }
I20260925 22:49:53.299139 10266 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd72b554da044868dde89586cea8fa0; no voters:
I20260925 22:49:53.299618 10266 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:53.299844 10271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:53.300117 10271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd72b554da044868dde89586cea8fa0, State: Running, Role: LEADER
I20260925 22:49:53.300586 10271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } }
I20260925 22:49:53.301291 10266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:53.302268 10272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd72b554da044868dde89586cea8fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } } }
I20260925 22:49:53.303447 10272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:53.304116 10273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd72b554da044868dde89586cea8fa0. Latest consensus state: current_term: 1 leader_uuid: "5bd72b554da044868dde89586cea8fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd72b554da044868dde89586cea8fa0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44221 } } }
I20260925 22:49:53.304134 10284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:53.304826 10273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:53.308257 10284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:53.314400 10284 catalog_manager.cc:1347] Generated new cluster ID: 91d504b7e6e14267ba207871521c9135
I20260925 22:49:53.314488 10284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:53.325403 10284 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:53.326754 10284 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:53.334390 10284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd72b554da044868dde89586cea8fa0: Generated new TSK 0
I20260925 22:49:53.335318 10284 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:49:53.358486 10296 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:49:53.358592 10296 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:49:53.359326 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:53.342906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.4.219.190:44221)
0925 22:49:53.343048 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:53.343053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:53.343304 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:49:53.343454 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:53.343458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:53.343517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:53.343639 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:49:53.343906 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.344778 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.345672 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.346025 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.346145 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.346478 (+ 333us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:49:53.358874 (+ 12396us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:49:53.359169 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38334: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:49:53.359391 10269 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38334: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 22:49:53.359454 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:53.342822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44221 (local address 127.0.0.1:38334)
0925 22:49:53.343206 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:53.343218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:53.343317 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:53.343670 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:53.343673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:53.343679 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:49:53.343778 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:53.343783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.344905 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.344908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:53.345536 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:53.345543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.346197 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.346200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:53.346267 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.346291 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:49:53.359088 (+ 12797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:49:53.359247 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.219.190:44221: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33721/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":113,"threads_started":1}
I20260925 22:49:53.359892 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10199
2026-09-25T22:49:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (384 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:49:53.399276 4974 mini_oidc.cc:152] Starting JWKS server
I20260925 22:49:53.400156 4974 webserver.cc:460] Webserver started at https://127.0.0.1:39567/ using document root <none> and password file <none>
I20260925 22:49:53.400290 4974 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:49:53.400607 4974 webserver.cc:460] Webserver started at http://127.0.0.1:46247/ using document root <none> and password file <none>
2026-09-25T22:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:53Z Disabled control of system clock
I20260925 22:49:53.409441 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44995
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:41451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44995
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39567/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:49:53.524196 10302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:53.524499 10302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:53.524645 10302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:53.529088 10302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:53.529174 10302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:53.529191 10302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:53.529209 10302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:53.529225 10302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:49:53.529294 10302 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:49:53.529313 10302 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39567/jwks.json
I20260925 22:49:53.534610 10302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39567/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:53.535881 10302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:53.537447 10302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:53.546517 10307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.546607 10302 server_base.cc:1034] running on GCE node
W20260925 22:49:53.546517 10310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:53.546813 10308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.547431 10302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:53.548679 10302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:53.549889 10302 hybrid_clock.cc:648] HybridClock initialized: now 1790376593549859 us; error 41 us; skew 500 ppm
I20260925 22:49:53.551956 10302 webserver.cc:460] Webserver started at http://127.4.219.190:36361/ using document root <none> and password file <none>
I20260925 22:49:53.552695 10302 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:53.552793 10302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:53.553027 10302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:53.555186 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "55adde27e314472792ce0fc80d19c99b"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "aa1fb1360f26cf2fb2ce49fc3eba4c62"
server_key_iv: "f5dec02c3a9e3aff456d408edffdc9d1"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.555776 10302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "55adde27e314472792ce0fc80d19c99b"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "aa1fb1360f26cf2fb2ce49fc3eba4c62"
server_key_iv: "f5dec02c3a9e3aff456d408edffdc9d1"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.559616 10302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:53.562100 10316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:53.563519 10302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:53.563691 10302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "55adde27e314472792ce0fc80d19c99b"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "aa1fb1360f26cf2fb2ce49fc3eba4c62"
server_key_iv: "f5dec02c3a9e3aff456d408edffdc9d1"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.563814 10302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:53.583370 10302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:53.584136 10302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:53.584348 10302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:53.591455 10302 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44995
I20260925 22:49:53.591463 10368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44995 every 8 connection(s)
I20260925 22:49:53.592456 10302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:53.595100 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10302
I20260925 22:49:53.595250 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:53.595597 4974 external_mini_cluster.cc:1428] Setting key 80359b1c250ce50598e463d614906648
I20260925 22:49:53.595974 10369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:53.603667 10369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b: Bootstrap starting.
I20260925 22:49:53.606271 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:53.597120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.4.219.190:44995)
0925 22:49:53.597985 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:53.598002 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:53.598041 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:49:53.598761 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:53.598780 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:53.599241 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:53.599551 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:53.599960 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.601085 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.602349 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.603052 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.603202 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.604274 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:53.604279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:53.604295 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:53.604812 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:53.604876 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:53.604898 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:53.604982 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:53.605189 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:49:53.605511 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":150,"threads_started":1}
I20260925 22:49:53.606367 10369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:53.607271 10369 log.cc:824] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:53.609383 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:53.609820 10369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b: No bootstrap required, opened a new log
I20260925 22:49:53.614368 10369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } }
I20260925 22:49:53.614676 10369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:53.614766 10369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55adde27e314472792ce0fc80d19c99b, State: Initialized, Role: FOLLOWER
I20260925 22:49:53.615322 10369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } }
I20260925 22:49:53.615496 10369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:53.615564 10369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:53.615664 10369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:53.695609 10369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } }
I20260925 22:49:53.696134 10369 leader_election.cc:302] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55adde27e314472792ce0fc80d19c99b; no voters:
I20260925 22:49:53.696604 10369 leader_election.cc:288] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:53.696717 10374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:53.696969 10374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [term 1 LEADER]: Becoming Leader. State: Replica: 55adde27e314472792ce0fc80d19c99b, State: Running, Role: LEADER
I20260925 22:49:53.697351 10374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } }
I20260925 22:49:53.698066 10369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:53.699384 10376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55adde27e314472792ce0fc80d19c99b. Latest consensus state: current_term: 1 leader_uuid: "55adde27e314472792ce0fc80d19c99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } } }
I20260925 22:49:53.699402 10375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55adde27e314472792ce0fc80d19c99b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55adde27e314472792ce0fc80d19c99b" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44995 } } }
I20260925 22:49:53.699610 10375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:53.699558 10376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:53.700137 10383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:53.703205 10383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:53.708201 10383 catalog_manager.cc:1347] Generated new cluster ID: 74ec2bfb65f2494d8368961fd9e7c18d
I20260925 22:49:53.708268 10383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:53.728577 10383 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:53.729868 10383 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:53.736575 10383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55adde27e314472792ce0fc80d19c99b: Generated new TSK 0
I20260925 22:49:53.737411 10383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:53.762290 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:53.756025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.4.219.190:44995)
0925 22:49:53.756169 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:53.756174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:53.756338 (+ 164us) server_negotiation.cc:407] Connection header received
0925 22:49:53.756483 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:53.756487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:53.756575 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:53.756705 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:49:53.756938 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.757765 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.758698 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.759061 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.759163 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.759513 (+ 350us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:49:53.761892 (+ 2379us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:49:53.762136 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:49:53.762365 10372 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35032: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:49:53.763994 10398 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:53.755871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44995 (local address 127.0.0.1:35032)
0925 22:49:53.756244 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:53.756258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:53.756365 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:53.756726 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:53.756728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:53.756735 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:49:53.756834 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:53.756839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.757888 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.757891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:53.758582 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:53.758592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:53.759229 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:53.759233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:53.759304 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:53.759328 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:49:53.762085 (+ 2757us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:49:53.763824 (+ 1739us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.219.190:44995: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39567/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1528,"thread_start_us":114,"threads_started":1}
I20260925 22:49:53.764678 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10302
E20260925 22:49:53.773461 10399 webserver.cc:533] Webserver: sslize failed:
E20260925 22:49:53.773535 10399 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:49:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (404 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:53Z Disabled control of system clock
I20260925 22:49:53.812695 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41313
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41313
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/testchainca.cert with env {}
W20260925 22:49:53.929605 10403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:53.929917 10403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:53.929961 10403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:53.934119 10403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:53.934192 10403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:53.934209 10403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:53.934226 10403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:53.934242 10403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:49:53.934257 10403 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:49:53.934271 10403 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:49:53.939925 10403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:53.941082 10403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:53.942083 10403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:53.951311 10408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:53.951340 10409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.951381 10403 server_base.cc:1034] running on GCE node
W20260925 22:49:53.951311 10411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:53.952101 10403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:53.953332 10403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:53.954501 10403 hybrid_clock.cc:648] HybridClock initialized: now 1790376593954479 us; error 35 us; skew 500 ppm
I20260925 22:49:53.956434 10403 webserver.cc:460] Webserver started at http://127.4.219.190:41359/ using document root <none> and password file <none>
I20260925 22:49:53.957044 10403 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:53.957194 10403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:53.957451 10403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:53.959594 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "6779795dba16aa456d3c7f7bc89f478f"
server_key_iv: "d5c258fb386a8a258fe228c2e365fb9c"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.960162 10403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "6779795dba16aa456d3c7f7bc89f478f"
server_key_iv: "d5c258fb386a8a258fe228c2e365fb9c"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.963965 10403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:53.966666 10417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:53.968079 10403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:53.968247 10403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5"
format_stamp: "Formatted at 2026-09-25 22:49:53 on dist-test-slave-kb1g"
server_key: "6779795dba16aa456d3c7f7bc89f478f"
server_key_iv: "d5c258fb386a8a258fe228c2e365fb9c"
server_key_version: "encryptionkey@0"
I20260925 22:49:53.968386 10403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:53.983003 10403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:53.983796 10403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:53.984014 10403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:53.991601 10403 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41313
I20260925 22:49:53.991611 10469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41313 every 8 connection(s)
I20260925 22:49:53.992725 10403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:53.996150 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:53.998579 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10403
I20260925 22:49:53.998746 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:53.999037 4974 external_mini_cluster.cc:1428] Setting key 4d535377903c806f47165551e2b56da5
I20260925 22:49:54.002682 10470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5: Bootstrap starting.
I20260925 22:49:54.005749 10470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:54.006726 10470 log.cc:824] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:54.009087 10470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5: No bootstrap required, opened a new log
I20260925 22:49:54.012789 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:54.000619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.4.219.190:41313)
0925 22:49:54.003361 (+ 2742us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:54.003384 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:54.003431 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:49:54.004385 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:54.004411 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:54.005014 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:54.005432 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:54.005955 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.007582 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.008857 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.009578 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.009734 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:54.010719 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:54.010724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:54.010738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:54.011166 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:54.011233 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:54.011255 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:54.011340 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:54.011566 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 22:49:54.011913 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2251,"thread_start_us":144,"threads_started":1}
I20260925 22:49:54.013998 10470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } }
I20260925 22:49:54.014274 10470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:54.014356 10470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d3927f30eb4a958f94c6ac8f8bb6f5, State: Initialized, Role: FOLLOWER
I20260925 22:49:54.014883 10470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } }
I20260925 22:49:54.015062 10470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:54.015136 10470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:54.015244 10470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:54.015924 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:54.101873 10470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } }
I20260925 22:49:54.102409 10470 leader_election.cc:302] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83d3927f30eb4a958f94c6ac8f8bb6f5; no voters:
I20260925 22:49:54.102869 10470 leader_election.cc:288] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:54.103029 10475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:54.103323 10475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [term 1 LEADER]: Becoming Leader. State: Replica: 83d3927f30eb4a958f94c6ac8f8bb6f5, State: Running, Role: LEADER
I20260925 22:49:54.103706 10475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } }
I20260925 22:49:54.104569 10470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:54.105603 10476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } } }
I20260925 22:49:54.105770 10476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:54.106108 10477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83d3927f30eb4a958f94c6ac8f8bb6f5. Latest consensus state: current_term: 1 leader_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83d3927f30eb4a958f94c6ac8f8bb6f5" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41313 } } }
I20260925 22:49:54.106233 10484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:54.106259 10477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:54.109160 10484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:54.114399 10484 catalog_manager.cc:1347] Generated new cluster ID: 339d7536072f4264a1eb4e16ff8705e3
I20260925 22:49:54.114483 10484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:54.125995 10484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:54.126922 10484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:54.136518 10484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83d3927f30eb4a958f94c6ac8f8bb6f5: Generated new TSK 0
I20260925 22:49:54.137269 10484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:54.212030 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:54.155413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47632 (local address 127.4.219.190:41313)
0925 22:49:54.155567 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:54.155572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:54.155799 (+ 227us) server_negotiation.cc:407] Connection header received
0925 22:49:54.155981 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:54.155984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:54.156035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:54.156131 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:49:54.156392 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.157028 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.158267 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.158490 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.158573 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:54.158936 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:49:54.211611 (+ 52675us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:49:54.211907 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47632: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:49:54.212134 10473 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47632: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:49:54.212711 10499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:54.155345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41313 (local address 127.0.0.1:47632)
0925 22:49:54.155687 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:54.155703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:54.155831 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:54.156152 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:54.156155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:54.156164 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:49:54.156286 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:54.156293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.157132 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.157136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:54.158140 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:54.158151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:54.158619 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:54.158623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:54.158734 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:54.158763 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:49:54.211928 (+ 53165us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:49:54.212164 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.219.190:41313: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260925 22:49:54.213385 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10403
2026-09-25T22:49:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (428 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:54 dist-test-slave-kb1g krb5kdc[10511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:54 dist-test-slave-kb1g krb5kdc[10511](info): set up 2 sockets
Sep 25 22:49:54 dist-test-slave-kb1g krb5kdc[10511](info): commencing operation
krb5kdc: starting...
W20260925 22:49:55.496933 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:55 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:49:55.619551 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39535
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:55.729811 10524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:55.730058 10524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:55.730098 10524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:55.734088 10524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:55.734145 10524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:55.734166 10524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:55.734182 10524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:55.734197 10524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:55.739341 10524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:55.740373 10524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:55.741415 10524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:55.749963 10529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:55.750000 10530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:55.750082 10524 server_base.cc:1034] running on GCE node
W20260925 22:49:55.750005 10532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:55.750725 10524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:55.751780 10524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:55.752960 10524 hybrid_clock.cc:648] HybridClock initialized: now 1790376595752931 us; error 43 us; skew 500 ppm
Sep 25 22:49:55 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376595, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:55.761111 10524 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:49:55.762069 10524 webserver.cc:460] Webserver started at http://127.4.219.190:44435/ using document root <none> and password file <none>
I20260925 22:49:55.762627 10524 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:55.762724 10524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:55.762959 10524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:55.765048 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "1112f90e142c4611b45a19920137e474"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-kb1g"
server_key: "ef6a7df40a50f8b36d9c8efdc8c9bb9b"
server_key_iv: "95b0b6f24892c0ace06d94d53cb81f11"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.765602 10524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "1112f90e142c4611b45a19920137e474"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-kb1g"
server_key: "ef6a7df40a50f8b36d9c8efdc8c9bb9b"
server_key_iv: "95b0b6f24892c0ace06d94d53cb81f11"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.769220 10524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:55.771533 10539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:55.772795 10524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:55.772941 10524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "1112f90e142c4611b45a19920137e474"
format_stamp: "Formatted at 2026-09-25 22:49:55 on dist-test-slave-kb1g"
server_key: "ef6a7df40a50f8b36d9c8efdc8c9bb9b"
server_key_iv: "95b0b6f24892c0ace06d94d53cb81f11"
server_key_version: "encryptionkey@0"
I20260925 22:49:55.773047 10524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:55.796900 10524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:55.800084 10524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:55.800302 10524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:55.807804 10524 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39535
I20260925 22:49:55.807814 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39535 every 8 connection(s)
I20260925 22:49:55.808816 10524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:55.811672 10592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:55.815487 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10524
I20260925 22:49:55.815618 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:55.815888 4974 external_mini_cluster.cc:1428] Setting key c54057de207ad29947b6a4d7e2e391b1
I20260925 22:49:55.817597 10592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474: Bootstrap starting.
I20260925 22:49:55.819993 10592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:55.820762 10592 log.cc:824] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:55.822778 10592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474: No bootstrap required, opened a new log
Sep 25 22:49:55 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:55.826727 10592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } }
I20260925 22:49:55.826964 10592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:55.827054 10592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1112f90e142c4611b45a19920137e474, State: Initialized, Role: FOLLOWER
I20260925 22:49:55.827515 10592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } }
I20260925 22:49:55.827666 10592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:55.827744 10592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:55.827842 10592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:55.828956 10592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } }
I20260925 22:49:55.829327 10592 leader_election.cc:302] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1112f90e142c4611b45a19920137e474; no voters:
I20260925 22:49:55.829640 10592 leader_election.cc:288] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:55.829774 10597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:55.830030 10597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [term 1 LEADER]: Becoming Leader. State: Replica: 1112f90e142c4611b45a19920137e474, State: Running, Role: LEADER
I20260925 22:49:55.830360 10597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } }
I20260925 22:49:55.830482 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:55.817312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.4.219.190:39535)
0925 22:49:55.817702 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:55.817724 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:55.817771 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:49:55.818386 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:55.818402 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:55.818727 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:55.819006 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:55.819843 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:55.820967 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:55.822038 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:55.822506 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:55.822619 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:55.825427 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:55.825432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:55.825446 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:55.825471 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:55.827314 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:55.827747 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:55.827756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:55.827761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:55.827910 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:55.828171 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:55.828175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:55.828176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:55.828695 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:55.828898 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:55.829097 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:49:55.829400 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260925 22:49:55.830899 10592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:55.831532 10597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1112f90e142c4611b45a19920137e474. Latest consensus state: current_term: 1 leader_uuid: "1112f90e142c4611b45a19920137e474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } } }
I20260925 22:49:55.831662 10597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:55.831946 10599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1112f90e142c4611b45a19920137e474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1112f90e142c4611b45a19920137e474" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39535 } } }
I20260925 22:49:55.832027 10599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:55.832769 10604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:55.835911 10604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:55.841261 10604 catalog_manager.cc:1347] Generated new cluster ID: e7487bd807b149ffae4432f7146348ad
I20260925 22:49:55.841332 10604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:55.857231 10604 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:55.858506 10604 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:55.868518 10604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1112f90e142c4611b45a19920137e474: Generated new TSK 0
I20260925 22:49:55.869416 10604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:49:55.900183 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:56.010322 10618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:56.010607 10618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:56.010680 10618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:56.014746 10618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:56.014834 10618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:56.014923 10618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:56.020074 10618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:56.021255 10618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:56.022558 10618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:56.032308 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:56.032308 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:56.032308 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.032742 10618 server_base.cc:1034] running on GCE node
I20260925 22:49:56.033149 10618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:56.033751 10618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:56.034900 10618 hybrid_clock.cc:648] HybridClock initialized: now 1790376596034876 us; error 35 us; skew 500 ppm
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.037745 10618 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:49:56.038546 10618 webserver.cc:460] Webserver started at http://127.4.219.129:40941/ using document root <none> and password file <none>
I20260925 22:49:56.039070 10618 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:56.039125 10618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:56.039285 10618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:56.041350 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "f3c777484f5a4fe681927633432c2fd0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "da57ba7ccff3c096a4febe88ecb0b469"
server_key_iv: "2e396281b208a9ab56c9c5b83a3bd1f8"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.041859 10618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "f3c777484f5a4fe681927633432c2fd0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "da57ba7ccff3c096a4febe88ecb0b469"
server_key_iv: "2e396281b208a9ab56c9c5b83a3bd1f8"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.045552 10618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:56.047856 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.049134 10618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:49:56.049264 10618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "f3c777484f5a4fe681927633432c2fd0"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "da57ba7ccff3c096a4febe88ecb0b469"
server_key_iv: "2e396281b208a9ab56c9c5b83a3bd1f8"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.049372 10618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:56.065820 10618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:56.069243 10618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:56.069448 10618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:56.070045 10618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:56.070850 10618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:56.070917 10618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.070981 10618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:56.071023 10618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.080749 10618 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:37023
I20260925 22:49:56.080771 10746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:37023 every 8 connection(s)
I20260925 22:49:56.081640 10618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:56.086632 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10618
I20260925 22:49:56.086726 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:56.086987 4974 external_mini_cluster.cc:1428] Setting key f07d9056e5d9eabc8ed494a2c69a9e43
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:56.096776 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.083462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:40299 (local address 127.4.219.190:39535)
0925 22:49:56.083608 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.083613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.084510 (+ 897us) server_negotiation.cc:407] Connection header received
0925 22:49:56.085865 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.085868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.085913 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.085988 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.087370 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.088012 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.089334 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.089736 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.089857 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.092705 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.092711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.092717 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.092753 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.094535 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.095056 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.095059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.095061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.095157 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.095479 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.095484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.095487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.095751 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.095857 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.096379 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.096517 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:56.097535 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.083910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39535 (local address 127.4.219.129:40299)
0925 22:49:56.084357 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:56.084400 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:56.085676 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:56.086173 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:56.086180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:56.086517 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:56.087214 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.087230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.088151 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.088156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.089212 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.089221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.089909 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.089915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.090149 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.090776 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:56.090802 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:56.092472 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:56.094666 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.094670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.094683 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.094939 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.095245 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.095247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.095248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.095376 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.095851 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:56.095856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:56.096114 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 22:49:56.096267 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:49:56.096518 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260925 22:49:56.098872 10747 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39535
I20260925 22:49:56.099134 10747 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:56.099637 10747 heartbeater.cc:511] Master 127.4.219.190:39535 requested a full tablet report, sending...
I20260925 22:49:56.101475 10556 ts_manager.cc:194] Registered new tserver with Master: f3c777484f5a4fe681927633432c2fd0 (127.4.219.129:37023)
I20260925 22:49:56.102603 10556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:40299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:49:56.115615 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:56.224313 10752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:56.224627 10752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:56.224725 10752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:56.228806 10752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:56.228864 10752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:56.228971 10752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:56.234352 10752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:56.235432 10752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:56.236495 10752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:56.246255 10760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:56.246325 10757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.246384 10752 server_base.cc:1034] running on GCE node
W20260925 22:49:56.246631 10758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.247080 10752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:56.247740 10752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:56.248919 10752 hybrid_clock.cc:648] HybridClock initialized: now 1790376596248895 us; error 36 us; skew 500 ppm
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.251724 10752 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:49:56.252694 10752 webserver.cc:460] Webserver started at http://127.4.219.130:41109/ using document root <none> and password file <none>
I20260925 22:49:56.253330 10752 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:56.253417 10752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:56.253649 10752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:56.255672 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "f8a9ee979371464e99a86aa98d22a786"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "9b856016b451ccfc4f9777b541d220c5"
server_key_iv: "6928bf8f3511e921219d8229819265f3"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.256222 10752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "f8a9ee979371464e99a86aa98d22a786"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "9b856016b451ccfc4f9777b541d220c5"
server_key_iv: "6928bf8f3511e921219d8229819265f3"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.259917 10752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:49:56.262498 10767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.263806 10752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:56.263952 10752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "f8a9ee979371464e99a86aa98d22a786"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "9b856016b451ccfc4f9777b541d220c5"
server_key_iv: "6928bf8f3511e921219d8229819265f3"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.264066 10752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:56.276799 10752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:56.280458 10752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:56.280700 10752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:56.281354 10752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:56.282233 10752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:56.282305 10752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.282371 10752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:56.282419 10752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.291239 10752 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44891
I20260925 22:49:56.291294 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44891 every 8 connection(s)
I20260925 22:49:56.292181 10752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:56.301694 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10752
I20260925 22:49:56.301820 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:56.302116 4974 external_mini_cluster.cc:1428] Setting key b1af4a3c9e7be6d665bd5d9f6bf80aef
I20260925 22:49:56.305989 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.294063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:42289 (local address 127.4.219.190:39535)
0925 22:49:56.294211 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.294216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.295095 (+ 879us) server_negotiation.cc:407] Connection header received
0925 22:49:56.296514 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.296518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.296598 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.296711 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.297901 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.298437 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.299582 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.299811 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.299920 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.302390 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.302394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.302396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.302419 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.303988 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.304480 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.304485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.304487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.304612 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.304935 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.304941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.304944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.305129 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.305234 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.305715 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.305833 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:49:56.306797 10883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.294447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39535 (local address 127.4.219.130:42289)
0925 22:49:56.294906 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:56.294949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:56.296296 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:56.296891 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:56.296898 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:56.297253 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:56.297748 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.297763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.298571 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.298576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.299444 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.299452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.299959 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.299963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.300131 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.300730 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:56.300746 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:56.302222 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:56.304099 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.304105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.304120 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.304376 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.304696 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.304699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.304701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.304839 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.305232 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:56.305239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:56.305475 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 22:49:56.305675 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 22:49:56.305987 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260925 22:49:56.307904 10881 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39535
I20260925 22:49:56.308154 10881 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:56.308689 10881 heartbeater.cc:511] Master 127.4.219.190:39535 requested a full tablet report, sending...
I20260925 22:49:56.309725 10556 ts_manager.cc:194] Registered new tserver with Master: f8a9ee979371464e99a86aa98d22a786 (127.4.219.130:44891)
I20260925 22:49:56.310535 10556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:42289
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:49:56.331298 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:56.447677 10886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:56.447959 10886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:56.448004 10886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:56.452119 10886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:56.452176 10886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:56.452277 10886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:56.457563 10886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:56.458565 10886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:56.459553 10886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:56.469498 10891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.469830 10886 server_base.cc:1034] running on GCE node
W20260925 22:49:56.470017 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:56.470960 10892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:56.471508 10886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:56.472208 10886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:56.473397 10886 hybrid_clock.cc:648] HybridClock initialized: now 1790376596473382 us; error 34 us; skew 500 ppm
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.476212 10886 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:49:56.477227 10886 webserver.cc:460] Webserver started at http://127.4.219.131:35305/ using document root <none> and password file <none>
I20260925 22:49:56.477807 10886 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:56.477869 10886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:56.478081 10886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:56.480152 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "e0be4dc59a9e4d42be1245d325face62"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "fc38099a6c1ec4ef800af0baa622a96e"
server_key_iv: "125f84f2651f06a9723c1a7e11e72677"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.480829 10886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "e0be4dc59a9e4d42be1245d325face62"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "fc38099a6c1ec4ef800af0baa622a96e"
server_key_iv: "125f84f2651f06a9723c1a7e11e72677"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.484483 10886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:49:56.486918 10901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.488184 10886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:49:56.488318 10886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "e0be4dc59a9e4d42be1245d325face62"
format_stamp: "Formatted at 2026-09-25 22:49:56 on dist-test-slave-kb1g"
server_key: "fc38099a6c1ec4ef800af0baa622a96e"
server_key_iv: "125f84f2651f06a9723c1a7e11e72677"
server_key_version: "encryptionkey@0"
I20260925 22:49:56.488442 10886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:56.510304 10886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:56.513491 10886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:56.513705 10886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:56.514307 10886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:56.515141 10886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:56.515211 10886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.515283 10886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:56.515327 10886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:56.524317 10886 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:43479
I20260925 22:49:56.524338 11014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:43479 every 8 connection(s)
I20260925 22:49:56.525348 10886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:56.528304 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 10886
I20260925 22:49:56.528419 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:56.528728 4974 external_mini_cluster.cc:1428] Setting key d61223b04634eec5aa20da908c088344
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:49:56.540066 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.527226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:56703 (local address 127.4.219.190:39535)
0925 22:49:56.527365 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.527369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.528224 (+ 855us) server_negotiation.cc:407] Connection header received
0925 22:49:56.530066 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.530069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.530139 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.530208 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.531648 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.532257 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.533542 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.533750 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.533829 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.536350 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.536356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.536361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.536396 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.537999 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.538551 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.538554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.538556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.538641 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.538982 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.538986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.538988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.539179 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.539286 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.539776 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.539896 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:56.540979 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.527531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39535 (local address 127.4.219.131:56703)
0925 22:49:56.528039 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:56.528088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:56.529828 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:56.530377 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:56.530389 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:56.530887 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:56.531510 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.531526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.532370 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.532373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.533434 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:56.533444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.533873 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.533876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:56.534048 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.534586 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:56.534605 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:56.536167 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:56.538142 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.538148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.538163 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.538432 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.538740 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:56.538742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:56.538744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:56.538886 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:56.539283 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:56.539290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:56.539542 (+ 252us) client_negotiation.cc:769] Sending connection context
0925 22:49:56.539735 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:49:56.540087 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":110,"threads_started":1}
I20260925 22:49:56.542297 11015 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39535
I20260925 22:49:56.542559 11015 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:56.543076 11015 heartbeater.cc:511] Master 127.4.219.190:39535 requested a full tablet report, sending...
I20260925 22:49:56.544005 10556 ts_manager.cc:194] Registered new tserver with Master: e0be4dc59a9e4d42be1245d325face62 (127.4.219.131:43479)
I20260925 22:49:56.544513 10556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:56703
I20260925 22:49:56.554665 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/test-admin.keytab.
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.569022 4974 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:49:56.571691 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:56.570390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.4.219.129:37023)
0925 22:49:56.570695 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.570716 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.570739 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:49:56.570791 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.570796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.570980 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.571103 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.571545 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48914: BlockingRecv error: recv got EOF from 127.0.0.1:48914 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":131,"threads_started":1}
W20260925 22:49:56.571849 4974 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.572919 4974 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.573879 4974 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:56.584016 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.574731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.4.219.129:37023)
0925 22:49:56.574856 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.574861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.574959 (+ 98us) server_negotiation.cc:407] Connection header received
0925 22:49:56.575111 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.575113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.575152 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.575237 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.575757 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.576598 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.577450 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.577834 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.577953 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.580296 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.580306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.580319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.580342 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.582095 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.582550 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.582557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.582559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.582643 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.582940 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.582943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.582945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.583250 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.583482 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.583697 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.583856 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:49:56.587873 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:56.587043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.4.219.129:37023)
0925 22:49:56.587188 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.587191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.587203 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:56.587311 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.587313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.587347 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.587411 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.587779 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48932: BlockingRecv error: recv got EOF from 127.0.0.1:48932 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 22:49:56.587981 4974 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.589025 4974 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.589885 4974 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:56.597968 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.590661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48940 (local address 127.4.219.129:37023)
0925 22:49:56.590768 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.590771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.590901 (+ 130us) server_negotiation.cc:407] Connection header received
0925 22:49:56.590985 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.590987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.591019 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.591084 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.591548 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.592149 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.592950 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.593180 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.593257 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.595299 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.595302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.595305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.595327 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.596674 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.597029 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.597031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.597033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.597100 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.597337 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.597339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.597341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.597495 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.597587 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.597774 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.597859 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 22:49:56.600507 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:56.599544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.4.219.129:37023)
0925 22:49:56.599652 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.599656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.599803 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:49:56.599916 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.599919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.599963 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.600043 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.600405 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48952: BlockingRecv error: recv got EOF from 127.0.0.1:48952 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 22:49:56.600935 4974 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376566921023-4974-0/krb5kdc/krb5cc)
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.601908 4974 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:56.602747 4974 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[10511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:49:56.611003 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:56.603533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48956 (local address 127.4.219.129:37023)
0925 22:49:56.603661 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:56.603666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:56.603755 (+ 89us) server_negotiation.cc:407] Connection header received
0925 22:49:56.603896 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:56.603899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:56.603943 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:56.604024 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:56.604514 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.605047 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.605815 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:56.606032 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:56.606138 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:56.608183 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:56.608187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:56.608190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:56.608219 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:56.609437 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.609846 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.609850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.609852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.609933 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:56.610206 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:56.610211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:56.610213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:56.610394 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:56.610530 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:56.610764 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:49:56.610865 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:49:56.612088 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10618
I20260925 22:49:56.617740 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10752
I20260925 22:49:56.622881 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10886
I20260925 22:49:56.628368 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 10524
2026-09-25T22:49:56Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2166 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[11035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[11035](info): set up 2 sockets
Sep 25 22:49:56 dist-test-slave-kb1g krb5kdc[11035](info): commencing operation
krb5kdc: starting...
W20260925 22:49:57.664923 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:49:57 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab.
Entry for principal oryx/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab.
I20260925 22:49:57.785445 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:42701
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab
--principal=oryx/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:57.896256 11048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:57.896572 11048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:57.896706 11048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:57.900779 11048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:57.900835 11048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:57.900856 11048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:57.900871 11048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:57.900924 11048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:57.906275 11048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:42701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab
--principal=oryx/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:57.907346 11048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:57.908396 11048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:57.917167 11053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:57.917138 11056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:57.917205 11048 server_base.cc:1034] running on GCE node
W20260925 22:49:57.917150 11054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:57.918002 11048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:57.919054 11048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:57.920228 11048 hybrid_clock.cc:648] HybridClock initialized: now 1790376597920215 us; error 30 us; skew 500 ppm
Sep 25 22:49:57 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:57.923224 11048 init.cc:377] Logged in from keytab as oryx/127.4.219.190@KRBTEST.COM (short username oryx)
I20260925 22:49:57.924165 11048 webserver.cc:460] Webserver started at http://127.4.219.190:46047/ using document root <none> and password file <none>
I20260925 22:49:57.924785 11048 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:57.924851 11048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:57.925038 11048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:57.927281 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "1139fe2a7ab541548c18cd9cbf968566"
format_stamp: "Formatted at 2026-09-25 22:49:57 on dist-test-slave-kb1g"
server_key: "344ddaf1968283e642d66a3b70d2caf8"
server_key_iv: "97e23a88146d4046b5de71699009a894"
server_key_version: "encryptionkey@0"
I20260925 22:49:57.927847 11048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "1139fe2a7ab541548c18cd9cbf968566"
format_stamp: "Formatted at 2026-09-25 22:49:57 on dist-test-slave-kb1g"
server_key: "344ddaf1968283e642d66a3b70d2caf8"
server_key_iv: "97e23a88146d4046b5de71699009a894"
server_key_version: "encryptionkey@0"
I20260925 22:49:57.931972 11048 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:49:57.934777 11063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:57.936152 11048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:57.936308 11048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "1139fe2a7ab541548c18cd9cbf968566"
format_stamp: "Formatted at 2026-09-25 22:49:57 on dist-test-slave-kb1g"
server_key: "344ddaf1968283e642d66a3b70d2caf8"
server_key_iv: "97e23a88146d4046b5de71699009a894"
server_key_version: "encryptionkey@0"
I20260925 22:49:57.936429 11048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:57.947487 11048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:57.950879 11048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:57.951097 11048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:57.957906 11048 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:42701
I20260925 22:49:57.958182 11115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:42701 every 8 connection(s)
I20260925 22:49:57.958889 11048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:57.961305 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11048
I20260925 22:49:57.961445 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:49:57.961753 4974 external_mini_cluster.cc:1428] Setting key 1e67f0dbbca8a9cc68fc40115af8e0d2
I20260925 22:49:57.961985 11116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:57.969216 11116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Bootstrap starting.
Sep 25 22:49:57 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:57.971393 11116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:57.972270 11116 log.cc:824] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:57.974438 11116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: No bootstrap required, opened a new log
I20260925 22:49:57.977025 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:57.963165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.4.219.190:42701)
0925 22:49:57.963574 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:57.963593 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:57.963636 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:49:57.964397 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:57.964422 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:57.964898 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:57.965286 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:57.966026 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:57.967511 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:57.968414 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:57.969111 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:57.969282 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:57.972078 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:57.972084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:57.972097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:57.972125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:57.974086 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:57.974625 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:57.974633 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:57.974638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:57.974762 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:57.975073 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:57.975075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:57.975076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:57.975450 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:57.975613 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:57.975913 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:49:57.976233 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260925 22:49:57.978497 11116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } }
I20260925 22:49:57.978770 11116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:57.978825 11116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1139fe2a7ab541548c18cd9cbf968566, State: Initialized, Role: FOLLOWER
I20260925 22:49:57.979244 11116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } }
I20260925 22:49:57.979396 11116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:57.979480 11116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:57.979604 11116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:57.980451 11116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } }
I20260925 22:49:57.980834 11116 leader_election.cc:302] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1139fe2a7ab541548c18cd9cbf968566; no voters:
I20260925 22:49:57.981209 11116 leader_election.cc:288] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:57.981310 11121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:57.981532 11121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 LEADER]: Becoming Leader. State: Replica: 1139fe2a7ab541548c18cd9cbf968566, State: Running, Role: LEADER
I20260925 22:49:57.981871 11121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } }
I20260925 22:49:57.982380 11116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:57.983537 11122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } }
I20260925 22:49:57.983542 11123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139fe2a7ab541548c18cd9cbf968566. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } }
I20260925 22:49:57.983956 11122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:57.983995 11123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:57.984766 11132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:57.987453 11132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:57.993145 11132 catalog_manager.cc:1347] Generated new cluster ID: 71ed8328425d4a69af9fe177889797a5
I20260925 22:49:57.993214 11132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:58.002395 11132 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:58.003629 11132 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:58.015859 11132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Generated new TSK 0
I20260925 22:49:58.016804 11132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
I20260925 22:49:58.047468 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:58.161682 11142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:58.161929 11142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:58.161971 11142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:58.166047 11142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:58.166106 11142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:58.166183 11142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:58.171458 11142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:58.172679 11142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:58.173803 11142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:58.183421 11147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.183460 11150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.183430 11148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.183547 11142 server_base.cc:1034] running on GCE node
I20260925 22:49:58.184257 11142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:58.184965 11142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:58.186163 11142 hybrid_clock.cc:648] HybridClock initialized: now 1790376598186144 us; error 31 us; skew 500 ppm
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:58.189098 11142 init.cc:377] Logged in from keytab as oryx/127.4.219.129@KRBTEST.COM (short username oryx)
I20260925 22:49:58.189971 11142 webserver.cc:460] Webserver started at http://127.4.219.129:38121/ using document root <none> and password file <none>
I20260925 22:49:58.190531 11142 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:58.190598 11142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:58.190806 11142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:58.192893 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "3938514dc0324358adff8e83616fcebf"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "91e74eb5135edc121bb4497e0cc57c39"
server_key_iv: "69b6cd71dba880dc26b0571267f8719c"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.193451 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "3938514dc0324358adff8e83616fcebf"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "91e74eb5135edc121bb4497e0cc57c39"
server_key_iv: "69b6cd71dba880dc26b0571267f8719c"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.197054 11142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:58.199388 11157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.200740 11142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:58.200855 11142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "3938514dc0324358adff8e83616fcebf"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "91e74eb5135edc121bb4497e0cc57c39"
server_key_iv: "69b6cd71dba880dc26b0571267f8719c"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.200954 11142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:58.223419 11142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:58.226874 11142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:58.227087 11142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:58.227692 11142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:58.228520 11142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:58.228624 11142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.228686 11142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:58.228716 11142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.237392 11142 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:41391
I20260925 22:49:58.237411 11270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:41391 every 8 connection(s)
I20260925 22:49:58.238274 11142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:58.244272 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11142
I20260925 22:49:58.244428 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:49:58.244781 4974 external_mini_cluster.cc:1428] Setting key bbcd649f3974f638319e635426ef5613
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:58.253436 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.240145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43131 (local address 127.4.219.190:42701)
0925 22:49:58.240301 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.240306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.241485 (+ 1179us) server_negotiation.cc:407] Connection header received
0925 22:49:58.242358 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.242361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.242410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.242490 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.243720 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.244272 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.245852 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.246217 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.246319 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.249476 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.249479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.249482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.249504 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.251080 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.251758 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.251761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.251763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.251839 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.252255 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.252257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.252259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.252434 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.252508 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.253210 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.253313 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:49:58.254638 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.240624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:43131)
0925 22:49:58.241327 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.241369 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.242146 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.242683 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.242690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.243074 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.243570 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.243585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.244426 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.244430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.245725 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.245735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.246385 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.246389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.246606 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.247222 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:58.247248 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:58.249317 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:58.251202 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.251211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.251229 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.251625 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.251936 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.251942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.251946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.252147 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.252567 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:58.252575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:58.252902 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 22:49:58.253112 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:49:58.253474 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for oryx/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:49:58.255967 11271 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:49:58.256286 11271 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:58.256866 11271 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.130@KRBTEST.COM" created.
I20260925 22:49:58.258445 11080 ts_manager.cc:194] Registered new tserver with Master: 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:49:58.260095 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:43131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
I20260925 22:49:58.273604 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:58.384745 11276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:58.385005 11276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:58.385047 11276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:58.388942 11276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:58.388990 11276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:58.389046 11276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:58.394173 11276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:58.395105 11276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:58.396047 11276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:58.405467 11276 server_base.cc:1034] running on GCE node
W20260925 22:49:58.405483 11281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.405506 11282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.405710 11284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.406117 11276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:58.406697 11276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:58.407887 11276 hybrid_clock.cc:648] HybridClock initialized: now 1790376598407875 us; error 23 us; skew 500 ppm
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:58.410606 11276 init.cc:377] Logged in from keytab as oryx/127.4.219.130@KRBTEST.COM (short username oryx)
I20260925 22:49:58.411427 11276 webserver.cc:460] Webserver started at http://127.4.219.130:35425/ using document root <none> and password file <none>
I20260925 22:49:58.411970 11276 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:58.412060 11276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:58.412281 11276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:58.414281 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "aac412a347dc4554a9fbbaec501a734c"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "376ee167cb8f87adae46db6f8c95ac58"
server_key_iv: "8949ef9b21048d5019c84df71e09d64b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.414822 11276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "aac412a347dc4554a9fbbaec501a734c"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "376ee167cb8f87adae46db6f8c95ac58"
server_key_iv: "8949ef9b21048d5019c84df71e09d64b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.420270 11276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:49:58.423434 11291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.425141 11276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:58.425266 11276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "aac412a347dc4554a9fbbaec501a734c"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "376ee167cb8f87adae46db6f8c95ac58"
server_key_iv: "8949ef9b21048d5019c84df71e09d64b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.425384 11276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:58.445787 11276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:58.448920 11276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:58.449117 11276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:58.449715 11276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:58.450510 11276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:58.450554 11276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.450616 11276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:58.450647 11276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.458843 11276 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44083
I20260925 22:49:58.458911 11404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44083 every 8 connection(s)
I20260925 22:49:58.459757 11276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:58.469998 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11276
I20260925 22:49:58.470131 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:49:58.470403 4974 external_mini_cluster.cc:1428] Setting key 1d44cb4de1a5ad87846cf145a6bf8672
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:58.477007 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.461637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49041 (local address 127.4.219.190:42701)
0925 22:49:58.461770 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.461773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.462966 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 22:49:58.464301 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.464305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.464361 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.464417 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.465961 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.466869 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.468275 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.468653 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.468768 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.472297 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.472304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.472308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.472338 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.474090 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.474869 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.474873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.474874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.474958 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.475430 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.475436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.475441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.475745 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.475847 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.476641 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.476794 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:58.478901 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.462028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:49041)
0925 22:49:58.462769 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.462828 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.464025 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.464816 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.464822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.465180 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.465802 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.465820 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.467028 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.467032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.468155 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.468165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.468936 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.468939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.469079 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.469742 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:58.469766 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:58.472110 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:58.474315 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.474325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.474348 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.474748 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.475061 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.475063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.475065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.475301 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.475907 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:58.475918 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:58.476286 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 22:49:58.477206 (+ 920us) client_negotiation.cc:240] Negotiation successful
0925 22:49:58.477631 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":109,"threads_started":1}
I20260925 22:49:58.480320 11405 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:49:58.480633 11405 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:58.481166 11405 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:49:58.482393 11080 ts_manager.cc:194] Registered new tserver with Master: aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083)
I20260925 22:49:58.483134 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:49041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
I20260925 22:49:58.499940 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:58.614355 11410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:58.614631 11410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:58.614703 11410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:58.619789 11410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:58.619864 11410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:58.619998 11410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:58.629041 11410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:42701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:58.630316 11410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:58.631636 11410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:58.642414 11415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.642474 11418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.642601 11410 server_base.cc:1034] running on GCE node
W20260925 22:49:58.642423 11416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.643258 11410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:58.643903 11410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:58.645068 11410 hybrid_clock.cc:648] HybridClock initialized: now 1790376598645021 us; error 60 us; skew 500 ppm
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:58.647969 11410 init.cc:377] Logged in from keytab as oryx/127.4.219.131@KRBTEST.COM (short username oryx)
I20260925 22:49:58.649044 11410 webserver.cc:460] Webserver started at http://127.4.219.131:39607/ using document root <none> and password file <none>
I20260925 22:49:58.649611 11410 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:58.649676 11410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:58.649849 11410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:58.652019 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "0428600fdf920d8fa41cdb088b4738d2"
server_key_iv: "59d71ab6bc8c2c045be3f990f87e228b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.652639 11410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "0428600fdf920d8fa41cdb088b4738d2"
server_key_iv: "59d71ab6bc8c2c045be3f990f87e228b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.656858 11410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:58.659417 11425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.660876 11410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:58.661027 11410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "0428600fdf920d8fa41cdb088b4738d2"
server_key_iv: "59d71ab6bc8c2c045be3f990f87e228b"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.661151 11410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:58.686489 11410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:58.689775 11410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:58.689989 11410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:58.690616 11410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:58.691468 11410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:58.691537 11410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.691610 11410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:58.691653 11410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.700659 11410 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:46403
I20260925 22:49:58.700682 11538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:46403 every 8 connection(s)
I20260925 22:49:58.701534 11410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:58.707019 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11410
I20260925 22:49:58.707146 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:49:58.707432 4974 external_mini_cluster.cc:1428] Setting key 2e024a25f5b827a58e36f122a16d12f8
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:58.715691 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.703287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:43703 (local address 127.4.219.190:42701)
0925 22:49:58.703425 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.703428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.704609 (+ 1181us) server_negotiation.cc:407] Connection header received
0925 22:49:58.705588 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.705591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.705638 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.705716 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.706891 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.707676 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.709119 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.709406 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.709546 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.712299 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.712302 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.712304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.712328 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.713753 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.714210 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.714213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.714214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.714331 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.714653 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.714656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.714658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.714854 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.714935 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.715430 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.715571 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:58.716586 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.703690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:43703)
0925 22:49:58.704396 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.704452 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.705382 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.705875 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.705881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.706272 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.706743 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.706757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.707815 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.707819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.708990 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.708999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.709597 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.709632 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.709798 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.710439 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:58.710455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:58.712087 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:58.713856 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.713862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.713873 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.714123 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.714417 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.714419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.714421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.714554 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.714946 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:58.714951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:58.715196 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 22:49:58.715357 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:49:58.715629 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":135,"threads_started":1}
I20260925 22:49:58.717867 11539 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:49:58.718122 11539 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:58.718633 11539 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
I20260925 22:49:58.719643 11080 ts_manager.cc:194] Registered new tserver with Master: fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403)
I20260925 22:49:58.720271 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:43703
I20260925 22:49:58.721838 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.219.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.189.keytab.
Entry for principal oryx/127.4.219.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.189.keytab.
I20260925 22:49:58.749526 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:37741
--webserver_interface=127.4.219.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.189.keytab
--principal=oryx/127.4.219.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:42701,127.4.219.189:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:58.861416 11544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:58.861676 11544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:58.861716 11544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:58.865759 11544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:58.865819 11544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:58.865840 11544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:58.865856 11544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:58.865871 11544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:58.871109 11544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:42701,127.4.219.189:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.189.keytab
--principal=oryx/127.4.219.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.219.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
W20260925 22:49:58.871304 11544 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.219.190:42701,127.4.219.189:37741'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:49:58.872146 11544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:58.873194 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:58.881390 11552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.881474 11550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:58.881472 11549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:58.881794 11544 server_base.cc:1034] running on GCE node
I20260925 22:49:58.882375 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:58.883728 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:58.884879 11544 hybrid_clock.cc:648] HybridClock initialized: now 1790376598884864 us; error 31 us; skew 500 ppm
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:58.887666 11544 init.cc:377] Logged in from keytab as oryx/127.4.219.189@KRBTEST.COM (short username oryx)
I20260925 22:49:58.888572 11544 webserver.cc:460] Webserver started at http://127.4.219.189:41361/ using document root <none> and password file <none>
I20260925 22:49:58.889108 11544 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:58.889165 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:58.889326 11544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:58.891325 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/instance:
uuid: "91c8fb921d25467aa14ceeabb17b6338"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "e5de5708f05084add3859aac843afcc0"
server_key_iv: "576978d51774d8258b52320ecd4bf641"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.891853 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal/instance:
uuid: "91c8fb921d25467aa14ceeabb17b6338"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "e5de5708f05084add3859aac843afcc0"
server_key_iv: "576978d51774d8258b52320ecd4bf641"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.895413 11544 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:49:58.897801 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:58.899050 11544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:58.899173 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "91c8fb921d25467aa14ceeabb17b6338"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "e5de5708f05084add3859aac843afcc0"
server_key_iv: "576978d51774d8258b52320ecd4bf641"
server_key_version: "encryptionkey@0"
I20260925 22:49:58.899277 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:58.931969 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:58.935379 11544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:58.935591 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:58.942847 11544 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.189:37741
I20260925 22:49:58.943169 11611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.189:37741 every 8 connection(s)
I20260925 22:49:58.943938 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
I20260925 22:49:58.946059 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11544
I20260925 22:49:58.947575 11612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:58.952000 11612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.189@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:58.963627 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.954306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.4.219.190:42701)
0925 22:49:58.954454 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.954458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.954772 (+ 314us) server_negotiation.cc:407] Connection header received
0925 22:49:58.954913 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.954918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.954983 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.955086 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.955849 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.956461 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.957401 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.957785 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.957886 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.960015 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.960018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.960021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.960052 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.961708 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.962143 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.962148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.962150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.962232 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.962489 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.962494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.962495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.962729 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.962820 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.963369 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.963478 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:58.964244 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.954037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51486)
0925 22:49:58.954635 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.954683 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.954801 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.955122 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.955126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.955458 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.955679 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.955698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.956618 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.956624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.957287 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.957296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.957905 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.957913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.957992 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.958439 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:58.958451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:58.959878 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:58.961820 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.961827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.961832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.962044 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.962324 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.962332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.962333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.962403 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.962867 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:58.962872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:58.963152 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 22:49:58.963294 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:49:58.963455 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":175,"threads_started":1}
I20260925 22:49:58.964244 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.947743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.4.219.189:37741)
0925 22:49:58.948185 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.948202 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.948240 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:49:58.948949 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.948968 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.949374 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.949677 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.950616 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.951810 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.952993 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.953484 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.953667 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.956724 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.956729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.956738 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.956764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.960466 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.961308 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.961314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.961321 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.961448 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.961752 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.961755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.961757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.962235 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.962437 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.962868 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.963175 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260925 22:49:58.965735 11612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:58 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376598, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.189@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
I20260925 22:49:58.973884 11616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.966238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53454)
0925 22:49:58.966373 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.966388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.966453 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.966858 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.966861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.967062 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.967142 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.967147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.968180 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.968184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.968750 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.968758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.969167 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.969170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.969255 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.969712 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:58.969723 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:58.970857 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:58.972408 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.972414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.972417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.972679 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.972943 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:58.972946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:58.972948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:58.973026 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:58.973374 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:58.973376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:58.973454 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 22:49:58.973564 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:49:58.973700 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:49:58.973884 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.966360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.4.219.189:37741)
0925 22:49:58.966700 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.966703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.966712 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:49:58.966736 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.966738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.966768 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.966850 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.967243 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.968059 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.968851 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.969070 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.969171 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:58.970983 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:58.970991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:58.970993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:58.971013 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:58.972307 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.972765 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.972771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.972773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.972840 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:58.973106 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:58.973111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:58.973113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:58.973270 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:58.973385 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:58.973605 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 22:49:58.973751 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":285}
I20260925 22:49:58.976943 11612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Bootstrap starting.
I20260925 22:49:58.979175 11612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:58.979945 11612 log.cc:824] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:58.981820 11612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: No bootstrap required, opened a new log
I20260925 22:49:58.985136 11612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } }
I20260925 22:49:58.985338 11612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:58.985373 11612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c8fb921d25467aa14ceeabb17b6338, State: Initialized, Role: FOLLOWER
I20260925 22:49:58.985823 11612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } }
I20260925 22:49:58.986266 11618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } }
I20260925 22:49:58.986395 11618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:58.987012 11612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:49:58.992024 11629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:58.992131 11629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:58.994218 11544 master_runner.cc:428] Detected that this master 91c8fb921d25467aa14ceeabb17b6338 is joining an existing cluster
I20260925 22:49:58.994297 11544 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.219.189:37741
I20260925 22:49:59.004458 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.996806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.4.219.189:37741)
0925 22:49:58.997026 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.997031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.997316 (+ 285us) server_negotiation.cc:407] Connection header received
0925 22:49:58.997481 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.997484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.997524 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.997590 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.998138 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.998626 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.999412 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.999615 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.999716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.000879 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.000882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.000885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.000911 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.002590 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.003215 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.003219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.003221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.003298 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.003548 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.003553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.003555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.003724 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.003832 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.004114 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.004318 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":142}
I20260925 22:49:59.004469 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.996889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53466)
0925 22:49:58.997237 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.997251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.997370 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.997690 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.997692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.997917 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.998022 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.998027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.998731 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.998736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.999304 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.999311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.999723 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.999726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:58.999827 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.000345 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.000364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.000754 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.002734 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.002736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.002738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.003079 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.003383 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.003388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.003390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.003467 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.003847 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.003850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.003942 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.004073 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.004264 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":5,"thread_start_us":68,"threads_started":1}
I20260925 22:49:59.005682 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.996892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.4.219.190:42701)
0925 22:49:58.997117 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:58.997121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:58.997305 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:49:58.997428 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:58.997431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:58.997482 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:58.997564 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:58.998193 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.999011 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.000207 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.000579 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.000657 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.002044 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.002048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.002051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.002079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.003740 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.004332 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.004336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.004338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.004418 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.004758 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.004761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.004763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.004968 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.005048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.005387 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.005507 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260925 22:49:59.006171 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:58.996806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51490)
0925 22:49:58.997212 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:58.997225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:58.997341 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:58.997592 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:58.997595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:58.997884 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:58.998039 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:58.998047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:58.999154 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:58.999157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.000071 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.000081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.000807 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.000811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.000904 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.001312 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.001332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.001885 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.003862 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.003864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.003867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.004217 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.004523 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.004548 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.004550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.004644 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.005122 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.005125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.005210 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.005862 (+ 652us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.006002 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"spinlock_wait_cycles":8192,"thread_start_us":112,"threads_started":1}
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.190@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
I20260925 22:49:59.020424 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.010301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53468)
0925 22:49:59.010735 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.010794 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.010928 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.011245 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.011252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.011710 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.011839 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.011847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.012641 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.012644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.013644 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.013654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.014068 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.014071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.014179 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.014698 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.014712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.016397 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.018495 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.018505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.018512 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.018822 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.019115 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.019119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.019121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.019216 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.019746 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.019753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.019874 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.020025 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.020246 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":124,"threads_started":1}
I20260925 22:49:59.020423 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.010309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.4.219.189:37741)
0925 22:49:59.010452 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.010457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.010874 (+ 417us) server_negotiation.cc:407] Connection header received
0925 22:49:59.011082 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.011084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.011127 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.011194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.011949 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.012501 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.013769 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.013970 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.014051 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.016576 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.016581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.016584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.016610 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.018388 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.018924 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.018931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.018935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.019022 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.019310 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.019316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.019320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.019577 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.019680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.020124 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.020266 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:59.022305 11080 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.219.189:37741: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338"
member_type: NON_VOTER
last_known_addr {
host: "127.4.219.189"
port: 37741
}
attrs {
promote: true
}
}
dest_uuid: "1139fe2a7ab541548c18cd9cbf968566"
cas_config_opid_index: -1
I20260925 22:49:59.022962 11080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } }
I20260925 22:49:59.024953 11641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.025605 11586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:59.026839 11641 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.219.189:37741
WARNING: no policy specified for oryx/127.4.219.188@KRBTEST.COM; defaulting to no policy
I20260925 22:49:59.026949 11586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:49:59.027395 11544 master.cc:537] Master@127.4.219.189:37741 shutting down...
I20260925 22:49:59.027863 11642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.188@KRBTEST.COM" created.
W20260925 22:49:59.031744 11067 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 22:49:59.032115 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.032238 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
W20260925 22:49:59.032701 11067 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 -> Peer 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189:37741): Couldn't send request to peer 91c8fb921d25467aa14ceeabb17b6338. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:49:59.032819 11544 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:49:59.033070 11544 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:49:59.033144 11544 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: stopping tablet replica
I20260925 22:49:59.037266 11544 master.cc:552] Master@127.4.219.189:37741 shutdown complete.
I20260925 22:49:59.037329 11544 master_runner.cc:305] Clearing existing system tablet
I20260925 22:49:59.037544 11544 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.037609 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.039034 11544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:49:59.039855 11648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.040071 11544 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.040200 11544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:59.040236 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "91c8fb921d25467aa14ceeabb17b6338"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "e5de5708f05084add3859aac843afcc0"
server_key_iv: "576978d51774d8258b52320ecd4bf641"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.042116 11544 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:59.044752 11544 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:49:59.044848 11544 log.cc:1196] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:49:59.045065 11544 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Deleting consensus metadata
I20260925 22:49:59.045228 11544 master_runner.cc:315] Copying system tablet from 127.4.219.190:42701
I20260925 22:49:59.046413 11544 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: tablet copy: Beginning tablet copy session from remote peer at address 127.4.219.190:42701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.188.keytab.
Entry for principal oryx/127.4.219.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.188.keytab.
I20260925 22:49:59.048074 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:37407
--webserver_interface=127.4.219.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.188.keytab
--principal=oryx/127.4.219.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:59.055657 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.047057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.4.219.190:42701)
0925 22:49:59.047187 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.047192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.048764 (+ 1572us) server_negotiation.cc:407] Connection header received
0925 22:49:59.048949 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.048955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.049010 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.049062 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.049702 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.050224 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.051097 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.051317 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.051424 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.052593 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.052597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.052601 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.052629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.054068 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.054509 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.054513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.054515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.054594 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.054856 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.054860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.054862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.055040 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.055141 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.055385 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.055489 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:49:59.055685 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.047010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51496)
0925 22:49:59.048636 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.048657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.048847 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.049214 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.049219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.049428 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.049558 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.049565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.050333 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.050339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.050981 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.050990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.051428 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.051432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.051520 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.052043 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.052055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.052394 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.054207 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.054212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.054214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.054415 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.054686 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.054689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.054691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.054772 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.055131 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.055133 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.055232 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.055336 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.055483 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":210,"threads_started":1}
I20260925 22:49:59.056478 11100 tablet_copy_service.cc:140] P 1139fe2a7ab541548c18cd9cbf968566: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91c8fb921d25467aa14ceeabb17b6338 ({username='oryx', principal='oryx/127.4.219.189@KRBTEST.COM'} at 127.0.0.1:51496)
I20260925 22:49:59.056749 11100 tablet_copy_service.cc:161] P 1139fe2a7ab541548c18cd9cbf968566: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91c8fb921d25467aa14ceeabb17b6338 at {username='oryx', principal='oryx/127.4.219.189@KRBTEST.COM'} at 127.0.0.1:51496: session id = 91c8fb921d25467aa14ceeabb17b6338-00000000000000000000000000000000
I20260925 22:49:59.058175 11100 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:49:59.059640 11544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:49:59.061590 11544 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: tablet copy: Starting download of 0 data blocks...
I20260925 22:49:59.061792 11544 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: tablet copy: Starting download of 1 WAL segments...
I20260925 22:49:59.063347 11544 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:49:59.064981 11100 tablet_copy_service.cc:342] P 1139fe2a7ab541548c18cd9cbf968566: Request end of tablet copy session 91c8fb921d25467aa14ceeabb17b6338-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.219.189@KRBTEST.COM'} at 127.0.0.1:51496
I20260925 22:49:59.065097 11100 tablet_copy_service.cc:433] P 1139fe2a7ab541548c18cd9cbf968566: ending tablet copy session 91c8fb921d25467aa14ceeabb17b6338-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91c8fb921d25467aa14ceeabb17b6338
I20260925 22:49:59.072726 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.074540 11656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.074568 11659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.075050 11544 server_base.cc:1034] running on GCE node
W20260925 22:49:59.076695 11657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.076931 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.077230 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.078372 11544 hybrid_clock.cc:648] HybridClock initialized: now 1790376599078360 us; error 29 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.079977 11544 init.cc:377] Logged in from keytab as oryx/127.4.219.189@KRBTEST.COM (short username oryx)
I20260925 22:49:59.080423 11544 webserver.cc:460] Webserver started at http://127.4.219.189:44205/ using document root <none> and password file <none>
I20260925 22:49:59.080639 11544 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.080687 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.082007 11544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:59.083055 11666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.083287 11544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:59.083392 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "91c8fb921d25467aa14ceeabb17b6338"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "e5de5708f05084add3859aac843afcc0"
server_key_iv: "576978d51774d8258b52320ecd4bf641"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.083614 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.098979 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.101328 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.106232 11544 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.189:37741
I20260925 22:49:59.106947 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
I20260925 22:49:59.108712 11718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.189:37741 every 8 connection(s)
I20260925 22:49:59.108968 11719 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:49:59.109974 11719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Bootstrap starting.
I20260925 22:49:59.119879 11719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:59.120187 11719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: Bootstrap complete.
I20260925 22:49:59.120808 11719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } }
I20260925 22:49:59.120946 11719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91c8fb921d25467aa14ceeabb17b6338, State: Initialized, Role: LEARNER
I20260925 22:49:59.121062 11719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } }
I20260925 22:49:59.121407 11722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.121539 11719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:59.121609 11722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:49:59.125298 11733 catalog_manager.cc:1259] Loaded cluster ID: 71ed8328425d4a69af9fe177889797a5
I20260925 22:49:59.125432 11733 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: loading cluster ID for follower catalog manager: success
I20260925 22:49:59.127511 11733 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: acquiring CA information for follower catalog manager: success
I20260925 22:49:59.128988 11733 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:49:59.180266 11655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:59.180590 11655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:59.180685 11655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:59.185278 11655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:59.185338 11655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:59.185357 11655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:59.185374 11655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:59.185410 11655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:59.190810 11655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.188.keytab
--principal=oryx/127.4.219.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.219.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:59.191951 11655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:59.193048 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.201282 11738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.201316 11741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.201282 11739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.201514 11655 server_base.cc:1034] running on GCE node
I20260925 22:49:59.202111 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.203264 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.204411 11655 hybrid_clock.cc:648] HybridClock initialized: now 1790376599204379 us; error 43 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.207152 11655 init.cc:377] Logged in from keytab as oryx/127.4.219.188@KRBTEST.COM (short username oryx)
I20260925 22:49:59.207983 11655 webserver.cc:460] Webserver started at http://127.4.219.188:45279/ using document root <none> and password file <none>
I20260925 22:49:59.208582 11655 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.208673 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.208897 11655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:59.210948 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/instance:
uuid: "2696a8da053046319f8fa2df95694dc9"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-kb1g"
server_key: "8087bca73bec47cff547ed1ec4244e67"
server_key_iv: "668208f58206e8a78c2e42256f6a7d4a"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.211513 11655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal/instance:
uuid: "2696a8da053046319f8fa2df95694dc9"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-kb1g"
server_key: "8087bca73bec47cff547ed1ec4244e67"
server_key_iv: "668208f58206e8a78c2e42256f6a7d4a"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.215157 11655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260925 22:49:59.217617 11748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.218864 11655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:59.219002 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "2696a8da053046319f8fa2df95694dc9"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-kb1g"
server_key: "8087bca73bec47cff547ed1ec4244e67"
server_key_iv: "668208f58206e8a78c2e42256f6a7d4a"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.219131 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.230314 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.233553 11655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:59.233750 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.241523 11655 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.188:37407
I20260925 22:49:59.241784 11800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.188:37407 every 8 connection(s)
I20260925 22:49:59.242563 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
I20260925 22:49:59.245375 11801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:59.245378 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11655
I20260925 22:49:59.250098 11801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.188@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:59.260272 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.246958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.4.219.188:37407)
0925 22:49:59.247376 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.247392 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.247427 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:49:59.248309 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.248329 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.248759 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.249054 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.249962 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.251018 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.252134 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.252838 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.252998 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.255359 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.255364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.255373 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.255398 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.257213 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.257659 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.257664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.257667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.257765 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.258158 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.258160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.258161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.258510 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.258650 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.258971 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.259252 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260925 22:49:59.260710 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.252100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.4.219.190:42701)
0925 22:49:59.252235 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.252239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.252407 (+ 168us) server_negotiation.cc:407] Connection header received
0925 22:49:59.252575 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.252578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.252636 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.252700 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.253455 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.253957 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.254827 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.255109 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.255216 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.257395 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.257400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.257403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.257428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.259083 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.259502 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.259505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.259506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.259596 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.259819 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.259822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.259823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.260019 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.260126 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.260412 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.260552 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:59.260741 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.251854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51508)
0925 22:49:59.252286 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.252332 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.252446 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.252769 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.252775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.253223 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.253337 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.253342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.254097 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.254100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.254708 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.254716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.255251 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.255256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.255360 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.255843 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.255854 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.257281 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.259205 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.259211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.259216 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.259410 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.259683 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.259684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.259685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.259744 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.260124 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.260128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.260213 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.260326 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.260572 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"spinlock_wait_cycles":2816,"thread_start_us":155,"threads_started":1}
I20260925 22:49:59.261809 11801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:59.262615 11271 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.188@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
I20260925 22:49:59.270081 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.262292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53482)
0925 22:49:59.262392 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.262406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.262478 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.263252 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.263255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.263409 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.263510 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.263516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.264634 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.264636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.265148 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.265156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.265612 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.265615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.265684 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.266152 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.266163 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.267322 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.268920 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.268922 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.268923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.269098 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.269357 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.269359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.269360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.269426 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.269731 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.269733 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.269800 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.269874 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.269964 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 22:49:59.270140 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.262723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.4.219.189:37741)
0925 22:49:59.263002 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.263007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.263023 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:49:59.263072 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.263075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.263127 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.263217 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.263651 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.264502 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.265281 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.265515 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.265591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.267440 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.267442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.267444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.267467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.268817 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.269197 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.269200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.269201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.269278 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.269493 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.269495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.269496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.269659 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.269732 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.269924 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.270015 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
I20260925 22:49:59.270953 11801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.188@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
I20260925 22:49:59.278486 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.271421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35910)
0925 22:49:59.271533 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.271564 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.271634 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.271890 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.271892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.272011 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.272077 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.272081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.273230 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.273233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.273714 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.273721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.274146 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.274149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.274203 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.274602 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.274612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.275710 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.277235 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.277237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.277238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.277423 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.277653 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.277655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.277656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.277729 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.278075 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.278078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.278154 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.278248 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.278337 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 22:49:59.278509 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.271464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.4.219.188:37407)
0925 22:49:59.271726 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.271729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.271739 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:49:59.271766 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.271768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.271804 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.271868 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.272196 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.273130 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.273808 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.274057 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.274135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.275823 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.275826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.275828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.275847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.277138 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.277508 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.277511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.277512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.277574 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.277800 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.277802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.277803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.277947 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.278015 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.278297 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.278387 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260925 22:49:59.281862 11801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Bootstrap starting.
I20260925 22:49:59.284384 11801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:59.285159 11801 log.cc:824] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:59.286983 11801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: No bootstrap required, opened a new log
I20260925 22:49:59.289928 11801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } }
I20260925 22:49:59.290125 11801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:59.290163 11801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2696a8da053046319f8fa2df95694dc9, State: Initialized, Role: FOLLOWER
I20260925 22:49:59.290516 11801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } }
I20260925 22:49:59.290946 11807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } } }
I20260925 22:49:59.291085 11807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:59.291653 11801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:49:59.296520 11818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:59.296641 11818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:59.299687 11655 master_runner.cc:428] Detected that this master 2696a8da053046319f8fa2df95694dc9 is joining an existing cluster
I20260925 22:49:59.299754 11655 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.219.188:37407
I20260925 22:49:59.310796 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35922)
0925 22:49:59.303310 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.303322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.303420 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.303715 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.303718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.303883 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.303996 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.304003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.304957 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.304960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306025 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.306035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306682 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306775 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.307376 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.307395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.307731 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.309514 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.309516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.309518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.309704 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.309965 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.309967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.309968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.310038 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.310420 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.310421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.310490 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.310566 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.310659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":124,"thread_start_us":94,"threads_started":1}
I20260925 22:49:59.310918 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.4.219.188:37407)
0925 22:49:59.302713 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.302718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.303375 (+ 657us) server_negotiation.cc:407] Connection header received
0925 22:49:59.303505 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.303508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.303545 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.303593 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.304107 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.304839 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306157 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306460 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306524 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.307856 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.307860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.307863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.307887 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.309415 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.309790 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.309794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.309796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.309879 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.310125 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.310129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.310131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.310332 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.310418 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.310635 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.310741 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 22:49:59.313691 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11142
I20260925 22:49:59.319809 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:41391
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:42701
--webserver_port=38121
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:59.329700 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.4.219.190:42701)
0925 22:49:59.302536 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.302540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.302604 (+ 64us) server_negotiation.cc:407] Connection header received
0925 22:49:59.302755 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.302761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.302815 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.302867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.303569 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.304833 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306313 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306649 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.325995 (+ 19271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.326001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.326004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.326035 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.327764 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.328266 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.328270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.328272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.328356 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.328683 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.328687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.328689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.328940 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.329010 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.329395 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.329524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:49:59.330243 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51514)
0925 22:49:59.302513 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.302528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.302631 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.303022 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.303025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.303300 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.303435 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.303442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.305000 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.305003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306182 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.306193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306871 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306978 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.325219 (+ 18241us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.325238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.325773 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.327899 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.327902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.327904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.328150 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.328460 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.328462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.328464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.328582 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.329119 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.329122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.329220 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.329853 (+ 633us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.330005 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":90,"thread_start_us":95,"threads_started":1}
I20260925 22:49:59.331518 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.4.219.189:37741)
0925 22:49:59.302853 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.302857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.303004 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:49:59.303172 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.303174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.303229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.303300 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.303975 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.304839 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306163 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306508 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306583 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.326204 (+ 19621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.326212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.326218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.326256 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.328651 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.329296 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.329302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.329306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.329404 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.329765 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.329771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.329776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.330021 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.330127 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.331222 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.331368 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:49:59.331624 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.302031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53490)
0925 22:49:59.302909 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.302920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.303075 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.303424 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.303427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.303721 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.303844 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.303851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.304957 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.304960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306040 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.306046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.306713 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.306716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.306866 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.325362 (+ 18496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.325385 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.325960 (+ 575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.328800 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.328804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.328808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.329113 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.329520 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.329524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.329528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.329647 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.330899 (+ 1252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.330905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.331029 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.331157 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.331334 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":231,"thread_start_us":102,"threads_started":1}
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376597, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.190@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
I20260925 22:49:59.345160 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.335559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35938)
0925 22:49:59.335726 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.335740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.335811 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.336209 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.336213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.336406 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.336511 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.336517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.337723 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.337726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.338647 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.338659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.339214 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.339218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.339310 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.339828 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.339844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.341504 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.343423 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.343427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.343429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.343683 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.344059 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.344063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.344068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.344161 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.344605 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.344608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.344714 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.344828 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.344981 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 22:49:59.345162 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.335644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.4.219.188:37407)
0925 22:49:59.335949 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.335953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.335967 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:49:59.336034 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.336037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.336097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.336171 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.336785 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.337613 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.338765 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.339104 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.339203 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.341641 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.341645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.341648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.341674 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.343312 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.343878 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.343882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.343883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.343964 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.344248 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.344251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.344253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.344454 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.344576 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.344887 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.344996 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260925 22:49:59.346593 11080 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.219.188:37407: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2696a8da053046319f8fa2df95694dc9"
member_type: NON_VOTER
last_known_addr {
host: "127.4.219.188"
port: 37407
}
attrs {
promote: true
}
}
dest_uuid: "1139fe2a7ab541548c18cd9cbf968566"
cas_config_opid_index: 5
I20260925 22:49:59.347189 11080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
I20260925 22:49:59.348232 11642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2696a8da053046319f8fa2df95694dc9 (127.4.219.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.348879 11775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:59.349814 11642 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.219.188:37407
I20260925 22:49:59.349983 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.350071 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.350281 11775 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:49:59.351434 11641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:59.352391 11655 master.cc:537] Master@127.4.219.188:37407 shutting down...
I20260925 22:49:59.355753 11775 raft_consensus.cc:858] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
W20260925 22:49:59.355872 11775 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
I20260925 22:49:59.357262 11775 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.359143 11828 mvcc.cc:205] Tried to move back new op lower bound from 7333382545468616704 to 7333382545334616064. Current Snapshot: MvccSnapshot[applied={T|T < 7333382545370206208}]
I20260925 22:49:59.359313 11807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139fe2a7ab541548c18cd9cbf968566. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.359419 11807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:49:59.359505 11807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.359576 11807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:49:59.359370 11832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } }
I20260925 22:49:59.359704 11832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 22:49:59.364163 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.351001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53500)
0925 22:49:59.351855 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.351871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.351954 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.360072 (+ 8118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.360075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.360110 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:59.360222 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.360230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.361439 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.361443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.362710 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.362721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.363670 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.363674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.363778 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.363830 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.363922 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.363999 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770}
I20260925 22:49:59.365000 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.351333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.4.219.189:37741)
0925 22:49:59.359740 (+ 8407us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.359745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.359764 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:59.359821 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.359826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.359891 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.360333 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:59.360433 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.361291 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.362863 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.363535 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.363641 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.363787 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.364804 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.364886 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8307}
I20260925 22:49:59.365485 11693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 LEARNER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:49:59.366178 11642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:49:59.366566 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.366684 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.366669 11655 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 22:49:59.366937 11655 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 22:49:59.367034 11655 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: stopping tablet replica
W20260925 22:49:59.367950 11064 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 -> Peer 2696a8da053046319f8fa2df95694dc9 (127.4.219.188:37407): Couldn't send request to peer 2696a8da053046319f8fa2df95694dc9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:49:59.368261 11693 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2696a8da053046319f8fa2df95694dc9 (127.4.219.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.370638 11722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139fe2a7ab541548c18cd9cbf968566. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.370771 11722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:49:59.370884 11722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: NON_VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: true } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.370985 11722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:49:59.373769 11643 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: attempting to promote NON_VOTER 91c8fb921d25467aa14ceeabb17b6338 to VOTER
I20260925 22:49:59.374249 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
I20260925 22:49:59.375480 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.375184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35942)
0925 22:49:59.375316 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.375398 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:49:59.376169 11693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 LEARNER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:49:59.376227 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.376003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35952)
0925 22:49:59.376117 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.376168 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:49:59.376319 11064 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 -> Peer 2696a8da053046319f8fa2df95694dc9 (127.4.219.188:37407): Couldn't send request to peer 2696a8da053046319f8fa2df95694dc9. Status: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:49:59.376724 11640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 22:49:59.378301 11641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.378746 11693 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.379805 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.379917 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.380038 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.380115 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:59.380337 11836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.380452 11836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:59.382593 11655 master.cc:552] Master@127.4.219.188:37407 shutdown complete.
I20260925 22:49:59.382650 11655 master_runner.cc:305] Clearing existing system tablet
I20260925 22:49:59.382913 11655 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.382983 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.384366 11655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:59.385286 11838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.385548 11655 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.385668 11655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:49:59.385706 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "2696a8da053046319f8fa2df95694dc9"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-kb1g"
server_key: "8087bca73bec47cff547ed1ec4244e67"
server_key_iv: "668208f58206e8a78c2e42256f6a7d4a"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.387457 11655 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:49:59.389724 11655 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:49:59.389811 11655 log.cc:1196] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:49:59.390100 11655 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Deleting consensus metadata
I20260925 22:49:59.390250 11655 master_runner.cc:315] Copying system tablet from 127.4.219.190:42701
I20260925 22:49:59.391353 11655 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: tablet copy: Beginning tablet copy session from remote peer at address 127.4.219.190:42701
I20260925 22:49:59.401234 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.392018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.4.219.190:42701)
0925 22:49:59.392160 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.392165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.392434 (+ 269us) server_negotiation.cc:407] Connection header received
0925 22:49:59.392728 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.392732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.392788 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.392877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.393385 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.394212 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.395847 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.396201 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.396305 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.397647 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.397651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.397654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.397679 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.399306 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.399845 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.399849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.399850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.399954 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.400245 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.400249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.400250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.400454 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.400523 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.400950 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.401075 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:49:59.401975 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.392108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:51528)
0925 22:49:59.392340 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.392354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.392522 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.392923 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.392926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.393146 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.393285 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.393292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.394329 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.394333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.395719 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.395731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.396322 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.396326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.396421 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.397017 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.397032 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.397478 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.399427 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.399430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.399433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.399705 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.400064 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.400067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.400069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.400151 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.400637 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.400640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.400733 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.401676 (+ 943us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.401819 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":119,"threads_started":1}
I20260925 22:49:59.402652 11100 tablet_copy_service.cc:140] P 1139fe2a7ab541548c18cd9cbf968566: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2696a8da053046319f8fa2df95694dc9 ({username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:51528)
I20260925 22:49:59.402755 11100 tablet_copy_service.cc:161] P 1139fe2a7ab541548c18cd9cbf968566: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2696a8da053046319f8fa2df95694dc9 at {username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:51528: session id = 2696a8da053046319f8fa2df95694dc9-00000000000000000000000000000000
I20260925 22:49:59.403859 11100 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:49:59.405153 11655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:49:59.407518 11655 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: tablet copy: Starting download of 0 data blocks...
I20260925 22:49:59.407691 11655 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: tablet copy: Starting download of 1 WAL segments...
I20260925 22:49:59.409228 11655 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:49:59.410977 11100 tablet_copy_service.cc:342] P 1139fe2a7ab541548c18cd9cbf968566: Request end of tablet copy session 2696a8da053046319f8fa2df95694dc9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:51528
I20260925 22:49:59.411059 11100 tablet_copy_service.cc:433] P 1139fe2a7ab541548c18cd9cbf968566: ending tablet copy session 2696a8da053046319f8fa2df95694dc9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2696a8da053046319f8fa2df95694dc9
I20260925 22:49:59.418116 11655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.419806 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.420014 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.420276 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.420627 11655 server_base.cc:1034] running on GCE node
I20260925 22:49:59.420806 11655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.421175 11655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.422370 11655 hybrid_clock.cc:648] HybridClock initialized: now 1790376599422359 us; error 28 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.424165 11655 init.cc:377] Logged in from keytab as oryx/127.4.219.188@KRBTEST.COM (short username oryx)
I20260925 22:49:59.424703 11655 webserver.cc:460] Webserver started at http://127.4.219.188:40815/ using document root <none> and password file <none>
I20260925 22:49:59.424942 11655 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.425024 11655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.426438 11655 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:59.427482 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.427755 11655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:49:59.427829 11655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "2696a8da053046319f8fa2df95694dc9"
format_stamp: "Formatted at 2026-09-25 22:49:59 on dist-test-slave-kb1g"
server_key: "8087bca73bec47cff547ed1ec4244e67"
server_key_iv: "668208f58206e8a78c2e42256f6a7d4a"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.427910 11655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.448164 11655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.450435 11655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.455189 11655 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.188:37407
I20260925 22:49:59.455430 11907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.188:37407 every 8 connection(s)
I20260925 22:49:59.455787 11655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
I20260925 22:49:59.457595 11908 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:49:59.458738 11908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Bootstrap starting.
I20260925 22:49:59.467099 11908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:49:59.467408 11908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Bootstrap complete.
I20260925 22:49:59.467981 11908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
I20260925 22:49:59.468119 11908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2696a8da053046319f8fa2df95694dc9, State: Initialized, Role: LEARNER
I20260925 22:49:59.468241 11908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } }
I20260925 22:49:59.468616 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.468779 11908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:59.468798 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:49:59.471727 11922 catalog_manager.cc:1259] Loaded cluster ID: 71ed8328425d4a69af9fe177889797a5
I20260925 22:49:59.472816 11922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: loading cluster ID for follower catalog manager: success
I20260925 22:49:59.474938 11922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: acquiring CA information for follower catalog manager: success
I20260925 22:49:59.476107 11922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:49:59.478835 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:59.479178 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:59.479264 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:59.484767 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:59.484843 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:59.484958 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:49:59.485945 11405 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
I20260925 22:49:59.490835 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:41391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=38121
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:59.491942 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:59.493062 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.503263 11928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.503263 11927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.503393 11827 server_base.cc:1034] running on GCE node
W20260925 22:49:59.503289 11930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.504006 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.504722 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.505897 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790376599505879 us; error 30 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.508665 11827 init.cc:377] Logged in from keytab as oryx/127.4.219.129@KRBTEST.COM (short username oryx)
I20260925 22:49:59.509617 11827 webserver.cc:460] Webserver started at http://127.4.219.129:38121/ using document root <none> and password file <none>
I20260925 22:49:59.510205 11827 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.510305 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.514209 11827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:49:59.516503 11937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.517915 11827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:49:59.518059 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "3938514dc0324358adff8e83616fcebf"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "91e74eb5135edc121bb4497e0cc57c39"
server_key_iv: "69b6cd71dba880dc26b0571267f8719c"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.518612 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.542171 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.545462 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:59.545677 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.546276 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:59.547156 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:59.547224 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.547292 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:59.547338 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.556651 11827 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:41391
I20260925 22:49:59.556672 12050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:41391 every 8 connection(s)
I20260925 22:49:59.557767 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:49:59.560861 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 11827
I20260925 22:49:59.561056 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11276
I20260925 22:49:59.568441 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:44083
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:42701
--webserver_port=35425
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:49:59.578552 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:59701 (local address 127.4.219.190:42701)
0925 22:49:59.560186 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.560191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.564556 (+ 4365us) server_negotiation.cc:407] Connection header received
0925 22:49:59.564757 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.564763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.564826 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.564883 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.565657 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566576 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569209 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.569640 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569727 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.574892 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.574897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.574901 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.574934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.576504 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.577027 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.577030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.577032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.577117 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.577403 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.577407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.577409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.577618 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.577695 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.578253 (+ 558us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.578382 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:49:59.578562 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:53851 (local address 127.4.219.188:37407)
0925 22:49:59.561266 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.561271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.561805 (+ 534us) server_negotiation.cc:407] Connection header received
0925 22:49:59.563109 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.563116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.563183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.563251 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.565507 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566566 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.568885 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.569239 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569314 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.572810 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.572817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.572820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.572852 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.574499 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.576637 (+ 2138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.576641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.576643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.576738 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.577172 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.577176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.577178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.577424 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.577504 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.578253 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.578383 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":73,"threads_started":1}
I20260925 22:49:59.578639 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:39963 (local address 127.4.219.189:37741)
0925 22:49:59.561120 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.561124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.562567 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 22:49:59.563114 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.563121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.563190 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.563249 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.565508 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566332 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.568886 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.569238 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.573990 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.573997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.574003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.574048 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.575827 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.576612 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.576618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.576623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.576724 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.577200 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.577206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.577209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.577465 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.577560 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.578275 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.578426 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:49:59.580442 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:59701)
0925 22:49:59.564438 (+ 3850us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.564453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.564639 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.565049 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.565055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.565369 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.565517 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.565526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566717 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.566723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.569068 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.569082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.570590 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.570597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.570692 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.571210 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.571224 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.574732 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.576651 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.576654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.576656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.576910 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.577230 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.577232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.577234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.577305 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.577744 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.577747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.577961 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.578838 (+ 877us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.579211 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3757,"mutex_wait_us":1733,"spinlock_wait_cycles":10368,"thread_start_us":107,"threads_started":1}
I20260925 22:49:59.580458 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.4.219.129:39963)
0925 22:49:59.562452 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.562467 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.562850 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.563620 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.563627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.564358 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.565338 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.565357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566496 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.566499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.568744 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.568755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.569525 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.570087 (+ 558us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.570618 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.570633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.573811 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.575967 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.575972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.575976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.576449 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.576850 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.576856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.576860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.577075 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.577632 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.577638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.577964 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.578807 (+ 843us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.579310 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"mutex_wait_us":974,"spinlock_wait_cycles":18176,"thread_start_us":138,"threads_started":1}
I20260925 22:49:59.581164 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.560569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.4.219.129:53851)
0925 22:49:59.561631 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.561687 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.562843 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.563623 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.563630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.564358 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.565338 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.565357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.566691 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.566694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.568746 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.568755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.569525 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.569529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.569758 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.570486 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.570513 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.572618 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.574614 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.574620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.574637 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.576477 (+ 1840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.576853 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.576859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.576863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.577063 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.577581 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.577587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.577961 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.579252 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.579361 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":383,"spinlock_wait_cycles":14592,"thread_start_us":127,"threads_started":1}
I20260925 22:49:59.582213 12051 heartbeater.cc:347] Connected to a master server at 127.4.219.188:37407
I20260925 22:49:59.582387 12052 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:49:59.582491 12051 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.582490 12052 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.583125 12051 heartbeater.cc:511] Master 127.4.219.188:37407 requested a full tablet report, sending...
I20260925 22:49:59.583127 12052 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
I20260925 22:49:59.582213 12053 heartbeater.cc:347] Connected to a master server at 127.4.219.189:37741
I20260925 22:49:59.583847 12053 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.584004 12053 heartbeater.cc:511] Master 127.4.219.189:37741 requested a full tablet report, sending...
I20260925 22:49:59.584616 11080 ts_manager.cc:194] Re-registered known tserver with Master: 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:49:59.585139 11872 ts_manager.cc:194] Registered new tserver with Master: 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:49:59.585366 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:59701
I20260925 22:49:59.586206 11683 ts_manager.cc:194] Registered new tserver with Master: 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
W20260925 22:49:59.685922 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:59.686218 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:59.686302 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:59.693321 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:59.693404 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:59.693498 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:49:59.699132 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:44083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=35425
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:59.700300 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:59.701413 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.710927 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.710919 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.710901 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.710986 12059 server_base.cc:1034] running on GCE node
I20260925 22:49:59.711529 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.712141 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.713294 12059 hybrid_clock.cc:648] HybridClock initialized: now 1790376599713279 us; error 25 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.715898 12059 init.cc:377] Logged in from keytab as oryx/127.4.219.130@KRBTEST.COM (short username oryx)
I20260925 22:49:59.716796 12059 webserver.cc:460] Webserver started at http://127.4.219.130:35425/ using document root <none> and password file <none>
I20260925 22:49:59.717321 12059 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.717430 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.721136 12059 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:49:59.722442 11539 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
I20260925 22:49:59.723598 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.724879 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:59.724972 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "aac412a347dc4554a9fbbaec501a734c"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "376ee167cb8f87adae46db6f8c95ac58"
server_key_iv: "8949ef9b21048d5019c84df71e09d64b"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.725531 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.764822 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.767964 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:59.768124 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.768757 12059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:59.769762 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:59.769829 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.769897 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:59.769923 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.778802 12059 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44083
I20260925 22:49:59.778825 12187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44083 every 8 connection(s)
I20260925 22:49:59.779838 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:49:59.786499 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12059
I20260925 22:49:59.786686 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11410
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
I20260925 22:49:59.796123 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:46403
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:42701
--webserver_port=39607
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:59.798622 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.781609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:47689 (local address 127.4.219.189:37741)
0925 22:49:59.781923 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.781927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.783273 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 22:49:59.784057 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.784060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.784115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.784173 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.785781 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786379 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.788598 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.788993 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.794280 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.794283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.794285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.794310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.796010 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.796650 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.796656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.796659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.796747 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.797155 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.797161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.797165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.797427 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.797517 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.798262 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.798399 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260925 22:49:59.799187 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.782121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:55227 (local address 127.4.219.188:37407)
0925 22:49:59.782256 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.782261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.782798 (+ 537us) server_negotiation.cc:407] Connection header received
0925 22:49:59.784054 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.784058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.784126 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.784177 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.785805 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786606 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.788607 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.788986 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789055 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.795074 (+ 6019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.795081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.795087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.795122 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.796893 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.797399 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.797405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.797407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.797504 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.797979 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.797985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.797988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.798227 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.798343 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.798886 (+ 543us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.799020 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:49:59.800307 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.781597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38669 (local address 127.4.219.190:42701)
0925 22:49:59.781795 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.781801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.783326 (+ 1525us) server_negotiation.cc:407] Connection header received
0925 22:49:59.784064 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.784067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.784126 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.784222 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.785804 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786623 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.788607 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.788952 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.794700 (+ 5646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:59.794705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:59.794711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:59.794746 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:59.798611 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.799073 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.799077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.799079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.799158 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:59.799458 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:59.799461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:59.799463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:59.799668 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:59.799750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.800068 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.800177 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:49:59.800393 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.782129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.4.219.130:47689)
0925 22:49:59.783162 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.783175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.783799 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.784401 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.784410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.784995 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.785643 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.785658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786547 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.786550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.788428 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.788443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.789290 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.789591 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.790712 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.790732 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.794107 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.796138 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.796144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.796159 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.796482 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.796854 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.796856 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.796858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.797040 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.797607 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.797616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.797965 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.798842 (+ 877us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.799236 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":548,"spinlock_wait_cycles":71808,"thread_start_us":156,"threads_started":1}
I20260925 22:49:59.800396 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.782129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.4.219.130:55227)
0925 22:49:59.782649 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.782700 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.783799 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.784401 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.784410 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.784994 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.785650 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.785659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786755 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.786758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.788439 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.788449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.789264 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.789600 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.790551 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.790591 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.794907 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.797027 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.797033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.797038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.797288 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.797623 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.797626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.797628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.797704 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.798589 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.798592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.798695 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.799386 (+ 691us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.799547 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":540,"spinlock_wait_cycles":25088,"thread_start_us":154,"threads_started":1}
I20260925 22:49:59.800810 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.782129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:38669)
0925 22:49:59.783202 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.783215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.783806 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.784400 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.784409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.784980 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.785650 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.785659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.786755 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.786758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.788439 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.788449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.789264 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.789268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.789596 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.790574 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.790592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:59.794537 (+ 3945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:59.798719 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.798722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.798724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.798950 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.799266 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:59.799270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:59.799272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:59.799355 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:59.799803 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:59.799805 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:59.799882 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.800487 (+ 605us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.800647 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":1031,"spinlock_wait_cycles":42112,"thread_start_us":110,"threads_started":1}
I20260925 22:49:59.801700 12188 heartbeater.cc:347] Connected to a master server at 127.4.219.188:37407
I20260925 22:49:59.801700 12189 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:49:59.801826 12190 heartbeater.cc:347] Connected to a master server at 127.4.219.189:37741
I20260925 22:49:59.801985 12189 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.801985 12190 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.802433 12188 heartbeater.cc:464] Registering TS with master...
I20260925 22:49:59.802580 12189 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
I20260925 22:49:59.802595 12188 heartbeater.cc:511] Master 127.4.219.188:37407 requested a full tablet report, sending...
I20260925 22:49:59.802582 12190 heartbeater.cc:511] Master 127.4.219.189:37741 requested a full tablet report, sending...
I20260925 22:49:59.803683 11080 ts_manager.cc:194] Re-registered known tserver with Master: aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083)
I20260925 22:49:59.803755 11872 ts_manager.cc:194] Registered new tserver with Master: aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083)
I20260925 22:49:59.803858 11683 ts_manager.cc:194] Registered new tserver with Master: aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083)
I20260925 22:49:59.804423 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:38669
I20260925 22:49:59.885339 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.881018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.4.219.188:37407)
0925 22:49:59.881140 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.881145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.881355 (+ 210us) server_negotiation.cc:407] Connection header received
0925 22:49:59.881596 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.881599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.881656 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.881749 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:49:59.882046 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.882666 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.884267 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.884834 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.884925 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.885052 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:59.885183 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:49:59.885237 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:59.885402 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.880953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:35968)
0925 22:49:59.881238 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.881257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.881412 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.881785 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.881788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.881811 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:49:59.881923 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.881929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.882768 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.882773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.884134 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.884146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.884950 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.884956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.885055 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.885084 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:49:59.885173 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:49:59.885246 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260925 22:49:59.886132 11882 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:49:59.886382 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139fe2a7ab541548c18cd9cbf968566. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:49:59.886541 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: LEARNER
W20260925 22:49:59.910843 12195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:59.911113 12195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:59.911186 12195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:59.915218 12195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:59.915297 12195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:59.915385 12195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:49:59.920583 12195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:46403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=39607
--tserver_master_addrs=127.4.219.190:42701,127.4.219.189:37741,127.4.219.188:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:59.921689 12195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:59.922832 12195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:59.932404 12204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:59.932523 12201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.932622 12195 server_base.cc:1034] running on GCE node
W20260925 22:49:59.932404 12202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:59.933193 12195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:59.933789 12195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:59.934944 12195 hybrid_clock.cc:648] HybridClock initialized: now 1790376599934929 us; error 27 us; skew 500 ppm
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:59.937762 12195 init.cc:377] Logged in from keytab as oryx/127.4.219.131@KRBTEST.COM (short username oryx)
I20260925 22:49:59.938612 12195 webserver.cc:460] Webserver started at http://127.4.219.131:39607/ using document root <none> and password file <none>
I20260925 22:49:59.939175 12195 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:59.939276 12195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:59.943397 12195 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 22:49:59.945748 12211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.947021 12195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:49:59.947160 12195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a"
format_stamp: "Formatted at 2026-09-25 22:49:58 on dist-test-slave-kb1g"
server_key: "0428600fdf920d8fa41cdb088b4738d2"
server_key_iv: "59d71ab6bc8c2c045be3f990f87e228b"
server_key_version: "encryptionkey@0"
I20260925 22:49:59.947717 12195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:59.973407 12195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:59.976270 12195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:59.976470 12195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:59.977098 12195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:49:59.977953 12195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:49:59.978039 12195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.978109 12195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:49:59.978149 12195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:59.987293 12195 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:46403
I20260925 22:49:59.987318 12324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:46403 every 8 connection(s)
I20260925 22:49:59.988334 12195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:49:59.995287 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12195
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.189@KRBTEST.COM
Sep 25 22:49:59 dist-test-slave-kb1g krb5kdc[11035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376599, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.188@KRBTEST.COM
I20260925 22:50:00.007133 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.990248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:58887 (local address 127.4.219.188:37407)
0925 22:49:59.990424 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.990430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.991729 (+ 1299us) server_negotiation.cc:407] Connection header received
0925 22:49:59.992809 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.992812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.992881 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.992984 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.994561 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995156 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997094 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997456 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997531 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.001863 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.001867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.001870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.001911 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.003697 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.005230 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.005234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.005237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.005339 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.005769 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.005773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.005775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.005995 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.006105 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.006803 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.006961 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:50:00.008582 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.990823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.4.219.131:58887)
0925 22:49:59.991579 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.991632 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.992587 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.993200 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.993212 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.993836 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.994417 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.994432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995325 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.995329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.996958 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.996968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997700 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.998820 (+ 1116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.999654 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.999669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:00.001676 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:00.003841 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.003850 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.003869 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.004231 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.005469 (+ 1238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.005475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.005480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.005670 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.006166 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:00.006172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:00.006524 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.006750 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.007113 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":1168,"spinlock_wait_cycles":10496,"thread_start_us":141,"threads_started":1}
I20260925 22:50:00.010088 12325 heartbeater.cc:347] Connected to a master server at 127.4.219.188:37407
I20260925 22:50:00.010387 12325 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:00.010588 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.991797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:48047 (local address 127.4.219.189:37741)
0925 22:49:59.991929 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.991933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.992009 (+ 76us) server_negotiation.cc:407] Connection header received
0925 22:49:59.993390 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.993395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.993476 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.993566 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.994581 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995400 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997066 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997393 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.005690 (+ 8224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.005697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.005700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.005731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.008181 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.008772 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.008776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.008777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.008856 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.009178 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.009181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.009183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.009374 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.009466 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.010272 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.010408 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:50:00.011008 12325 heartbeater.cc:511] Master 127.4.219.188:37407 requested a full tablet report, sending...
I20260925 22:50:00.010135 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.990823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.4.219.131:48047)
0925 22:49:59.991933 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.991944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.992600 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.993597 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.993602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.993840 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.994426 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.994436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995522 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.995524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.996922 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.996932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997663 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.998550 (+ 883us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.999019 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.999033 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:00.005556 (+ 6523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:00.008306 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.008311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.008314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.008662 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.008975 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.008980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.008984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.009081 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.009512 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:00.009516 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:00.009632 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.009734 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.009895 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":1012,"spinlock_wait_cycles":7936,"thread_start_us":119,"threads_started":1}
I20260925 22:50:00.012387 11872 ts_manager.cc:194] Registered new tserver with Master: fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403)
I20260925 22:50:00.012773 12327 heartbeater.cc:347] Connected to a master server at 127.4.219.189:37741
I20260925 22:50:00.012862 12327 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:00.013027 12327 heartbeater.cc:511] Master 127.4.219.189:37741 requested a full tablet report, sending...
I20260925 22:50:00.013247 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.991212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45103 (local address 127.4.219.190:42701)
0925 22:49:59.991329 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.991333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.992299 (+ 966us) server_negotiation.cc:407] Connection header received
0925 22:49:59.993128 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.993131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.993183 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.993260 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:59.994588 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995365 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997060 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997392 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.005791 (+ 8325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.005795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.005797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.005827 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.008055 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.008671 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.008677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.008680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.008767 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.011778 (+ 3011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.011781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.011783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.012026 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.012131 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.012970 (+ 839us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.013087 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:00.013706 11683 ts_manager.cc:194] Registered new tserver with Master: fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403)
I20260925 22:50:00.013876 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.991078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:45103)
0925 22:49:59.992207 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:59.992217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:59.992587 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:59.993309 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:59.993312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:59.993839 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:59.994428 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.994436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.995511 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.995514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.996925 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:59.996932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:59.997663 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:59.997667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:59.997937 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:59.998712 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:59.998740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:00.005663 (+ 6923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:00.008191 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.008196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.008198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.008524 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.011596 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:00.011598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:00.011600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:00.011672 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:00.012369 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:00.012372 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:00.012468 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.012584 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.012744 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":338,"spinlock_wait_cycles":896,"thread_start_us":76,"threads_started":1}
I20260925 22:50:00.013983 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.998584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.4.219.189:37741)
0925 22:49:59.999390 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.999396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:59.999745 (+ 349us) server_negotiation.cc:407] Connection header received
0925 22:49:59.999789 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:59.999795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:59.999845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:59.999923 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.001369 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.002176 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.003557 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.003919 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.004014 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.009047 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.009051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.009054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.009086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.010805 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.011329 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.011333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.011334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.011426 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.012763 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.012766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.012768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.012999 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.013120 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.013671 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.013799 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":78,"threads_started":1}
I20260925 22:50:00.014374 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.998085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.4.219.190:42701)
0925 22:49:59.998321 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:59.998325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.000314 (+ 1989us) server_negotiation.cc:407] Connection header received
0925 22:50:00.000417 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.000420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.000488 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.000694 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.001716 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.002561 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.003895 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.004252 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.004355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.010370 (+ 6015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.010375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.010378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.010403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.012390 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.012874 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.012877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.012879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.012955 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.013249 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.013253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.013254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.013485 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.013601 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.014093 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.014205 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260925 22:50:00.015035 12326 heartbeater.cc:347] Connected to a master server at 127.4.219.190:42701
I20260925 22:50:00.015108 12326 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:00.015268 12326 heartbeater.cc:511] Master 127.4.219.190:42701 requested a full tablet report, sending...
I20260925 22:50:00.016026 11080 ts_manager.cc:194] Re-registered known tserver with Master: fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403)
I20260925 22:50:00.016742 11080 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:45103
I20260925 22:50:00.016793 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:59.997481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.4.219.188:37407)
0925 22:50:00.000306 (+ 2825us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.000310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.000322 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:50:00.000360 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.000364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.000418 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.000500 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.001369 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.002162 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.003563 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.003913 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.004014 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.006183 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.006190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.006194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.006222 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.008231 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.015402 (+ 7171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.015405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.015407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.015545 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.015810 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.015813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.015816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.016035 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.016163 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.016467 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.016626 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2736,"thread_start_us":91,"threads_started":1}
I20260925 22:50:00.019672 11079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:00.022845 11079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:00.025312 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: NON_VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: true } } }
I20260925 22:50:00.025506 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:00.027024 12343 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: attempting to promote NON_VOTER 2696a8da053046319f8fa2df95694dc9 to VOTER
I20260925 22:50:00.027356 12343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } }
I20260925 22:50:00.028837 11693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:50:00.029031 11882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 LEARNER]: Refusing update from remote peer 1139fe2a7ab541548c18cd9cbf968566: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:50:00.029245 12343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:50:00.029381 11641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 22:50:00.030875 11641 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2696a8da053046319f8fa2df95694dc9 (127.4.219.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.031533 11693 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2696a8da053046319f8fa2df95694dc9 (127.4.219.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.031626 11882 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 2696a8da053046319f8fa2df95694dc9 (127.4.219.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.032311 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.032402 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:00.032521 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.032718 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:00.032855 11641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.032933 11641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:00.033106 11911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.033234 11911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:00.033732 12352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1139fe2a7ab541548c18cd9cbf968566" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:00.033859 12352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:00.043406 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.038111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41391 (local address 127.0.0.1:51610)
0925 22:50:00.038564 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:00.038577 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:00.038692 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:00.039202 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:00.039205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:00.039221 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:00.039311 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.039317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.040791 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.040794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.041901 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.041912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.042983 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.042987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.043074 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.043104 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.043165 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.043236 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":99,"threads_started":1}
I20260925 22:50:00.043617 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.038709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:46403 (local address 127.0.0.1:36198)
0925 22:50:00.038912 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:00.038924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:00.038996 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:00.040153 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:00.040157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:00.040175 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:00.040276 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.040282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.041387 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.041393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.042141 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.042148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.043093 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.043097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.043178 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.043200 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.043262 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.043339 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260925 22:50:00.043658 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.039121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.4.219.131:46403)
0925 22:50:00.039695 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.039705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.039726 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:50:00.039792 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.039797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.040019 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.040157 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:00.040369 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.041288 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.042264 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.042969 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.043093 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.043243 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.043293 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.043379 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":97,"threads_started":1}
I20260925 22:50:00.043825 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.037840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44083 (local address 127.0.0.1:39054)
0925 22:50:00.037954 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:00.037972 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:00.038054 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:00.038895 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:00.038900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:00.038918 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:00.039028 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.039035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.040608 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.040610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.041832 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.041841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.043422 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.043439 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.043550 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.043574 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.043644 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.043712 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 22:50:00.044119 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.038262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.4.219.129:41391)
0925 22:50:00.038493 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.038509 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.038672 (+ 163us) server_negotiation.cc:407] Connection header received
0925 22:50:00.038815 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.038821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.039061 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.039224 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:00.039516 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.040674 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.042048 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.042829 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.043641 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.043782 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.043835 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.043947 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260925 22:50:00.045202 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.037986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.4.219.130:44083)
0925 22:50:00.038353 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.038366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.038392 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:50:00.038471 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.038477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.038727 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.039188 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:00.039335 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.040449 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.041972 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.043298 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.044134 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.044927 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.044978 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.045067 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":140,"threads_started":1}
I20260925 22:50:00.046460 12259 tablet_service.cc:1467] Processing CreateTablet for tablet 6e520431e38841fca43dd5cb7083712d (DEFAULT_TABLE table=test-table [id=697d8cdf563c44b6a91b5559b95bc982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:00.046829 11985 tablet_service.cc:1467] Processing CreateTablet for tablet 6e520431e38841fca43dd5cb7083712d (DEFAULT_TABLE table=test-table [id=697d8cdf563c44b6a91b5559b95bc982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:00.047466 12259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e520431e38841fca43dd5cb7083712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:00.047855 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e520431e38841fca43dd5cb7083712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:00.048578 12122 tablet_service.cc:1467] Processing CreateTablet for tablet 6e520431e38841fca43dd5cb7083712d (DEFAULT_TABLE table=test-table [id=697d8cdf563c44b6a91b5559b95bc982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:00.049548 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e520431e38841fca43dd5cb7083712d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:00.054296 12358 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Bootstrap starting.
I20260925 22:50:00.055655 12359 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Bootstrap starting.
I20260925 22:50:00.056872 12358 tablet_bootstrap.cc:654] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:00.057709 12359 tablet_bootstrap.cc:654] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:00.057741 12358 log.cc:824] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:00.058495 12359 log.cc:824] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:00.058521 12361 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Bootstrap starting.
I20260925 22:50:00.059665 12358 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: No bootstrap required, opened a new log
I20260925 22:50:00.059867 12358 ts_tablet_manager.cc:1397] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:50:00.060376 12359 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: No bootstrap required, opened a new log
I20260925 22:50:00.060600 12359 ts_tablet_manager.cc:1397] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:50:00.060648 12361 tablet_bootstrap.cc:654] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:00.061408 12361 log.cc:824] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:00.063395 12361 tablet_bootstrap.cc:492] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: No bootstrap required, opened a new log
I20260925 22:50:00.063580 12361 ts_tablet_manager.cc:1397] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 22:50:00.064064 12358 raft_consensus.cc:348] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.064337 12358 raft_consensus.cc:374] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:00.064394 12358 raft_consensus.cc:729] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc9eebbb4ea4e03beb418e9d4f09c0a, State: Initialized, Role: FOLLOWER
I20260925 22:50:00.064855 12358 consensus_queue.cc:260] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.064824 12359 raft_consensus.cc:348] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.065059 12359 raft_consensus.cc:374] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:00.065116 12359 raft_consensus.cc:729] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3938514dc0324358adff8e83616fcebf, State: Initialized, Role: FOLLOWER
I20260925 22:50:00.065590 12326 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
I20260925 22:50:00.065845 12358 ts_tablet_manager.cc:1428] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:50:00.066305 12359 consensus_queue.cc:260] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.067258 12359 ts_tablet_manager.cc:1428] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:50:00.068181 12361 raft_consensus.cc:348] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.068426 12361 raft_consensus.cc:374] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:00.068480 12361 raft_consensus.cc:729] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac412a347dc4554a9fbbaec501a734c, State: Initialized, Role: FOLLOWER
I20260925 22:50:00.068773 12052 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
I20260925 22:50:00.068898 12361 consensus_queue.cc:260] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.069814 12361 ts_tablet_manager.cc:1428] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:50:00.070734 12189 heartbeater.cc:503] Master 127.4.219.190:42701 was elected leader, sending a full tablet report...
I20260925 22:50:00.188452 12364 raft_consensus.cc:482] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:00.188747 12364 raft_consensus.cc:504] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.189909 12364 leader_election.cc:288] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083), 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:50:00.194880 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.190290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:57245 (local address 127.4.219.129:41391)
0925 22:50:00.190419 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.190425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.190464 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:50:00.190633 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.190637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.190684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.190766 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:00.191172 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.191791 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.193551 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.194164 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.194292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.194346 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.194671 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.194745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:50:00.194880 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.190447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:47801 (local address 127.4.219.130:44083)
0925 22:50:00.190597 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.190604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.190618 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:00.190671 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.190673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.190708 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.190790 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:00.191117 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.191791 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.193551 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.194164 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.194292 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.194346 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.194671 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.194745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:50:00.194938 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.190201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41391 (local address 127.4.219.131:57245)
0925 22:50:00.190385 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:00.190404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:00.190493 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:00.190777 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:00.190779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:00.190794 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:00.191007 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.191015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.191926 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.191929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.193425 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.193436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.194290 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.194294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.194486 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.194567 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.194646 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.194757 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":250}
I20260925 22:50:00.194941 12330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.190229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44083 (local address 127.4.219.131:47801)
0925 22:50:00.190417 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:00.190432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:00.190526 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:00.190799 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:00.190802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:00.190817 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:00.191004 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.191010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.191889 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.191896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.193425 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:00.193436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.194290 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.194294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:00.194460 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.194568 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:50:00.194646 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:50:00.194757 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":273}
I20260925 22:50:00.195605 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e520431e38841fca43dd5cb7083712d" candidate_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3938514dc0324358adff8e83616fcebf" is_pre_election: true
I20260925 22:50:00.195734 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e520431e38841fca43dd5cb7083712d" candidate_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac412a347dc4554a9fbbaec501a734c" is_pre_election: true
I20260925 22:50:00.195925 12005 raft_consensus.cc:2454] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc9eebbb4ea4e03beb418e9d4f09c0a in term 0.
I20260925 22:50:00.196063 12142 raft_consensus.cc:2454] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcc9eebbb4ea4e03beb418e9d4f09c0a in term 0.
I20260925 22:50:00.196420 12214 leader_election.cc:302] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3938514dc0324358adff8e83616fcebf, aac412a347dc4554a9fbbaec501a734c, fcc9eebbb4ea4e03beb418e9d4f09c0a; no voters:
I20260925 22:50:00.196730 12364 raft_consensus.cc:2781] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:00.196806 12364 raft_consensus.cc:482] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:00.196836 12364 raft_consensus.cc:3037] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:00.197890 12364 raft_consensus.cc:504] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.198239 12364 leader_election.cc:288] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [CANDIDATE]: Term 1 election: Requested vote from peers aac412a347dc4554a9fbbaec501a734c (127.4.219.130:44083), 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:50:00.198587 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e520431e38841fca43dd5cb7083712d" candidate_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3938514dc0324358adff8e83616fcebf"
I20260925 22:50:00.198688 12005 raft_consensus.cc:3037] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:00.199699 12005 raft_consensus.cc:2454] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc9eebbb4ea4e03beb418e9d4f09c0a in term 1.
I20260925 22:50:00.199981 12214 leader_election.cc:302] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3938514dc0324358adff8e83616fcebf, fcc9eebbb4ea4e03beb418e9d4f09c0a; no voters:
I20260925 22:50:00.199965 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e520431e38841fca43dd5cb7083712d" candidate_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aac412a347dc4554a9fbbaec501a734c"
I20260925 22:50:00.200098 12142 raft_consensus.cc:3037] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:00.200155 12364 raft_consensus.cc:2781] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:00.200352 12364 raft_consensus.cc:686] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 LEADER]: Becoming Leader. State: Replica: fcc9eebbb4ea4e03beb418e9d4f09c0a, State: Running, Role: LEADER
I20260925 22:50:00.200809 12364 consensus_queue.cc:237] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } }
I20260925 22:50:00.201397 12142 raft_consensus.cc:2454] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcc9eebbb4ea4e03beb418e9d4f09c0a in term 1.
I20260925 22:50:00.205036 11079 catalog_manager.cc:5495] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a reported cstate change: term changed from 0 to 1, leader changed from <none> to fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131). New cstate: current_term: 1 leader_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } health_report { overall_health: HEALTHY } } }
I20260925 22:50:00.237176 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.233838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.4.219.131:46403)
0925 22:50:00.233962 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.233965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.233979 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:00.234107 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.234109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.234163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.234255 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:00.234465 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.234955 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.235750 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.235957 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.236054 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.236280 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:00.236772 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:00.236864 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.236918 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.236965 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:50:00.240182 12328 tablet.cc:2369] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:50:00.243317 12142 raft_consensus.cc:1261] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Refusing update from remote peer fcc9eebbb4ea4e03beb418e9d4f09c0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:00.243319 12005 raft_consensus.cc:1261] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Refusing update from remote peer fcc9eebbb4ea4e03beb418e9d4f09c0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:00.244136 12364 consensus_queue.cc:1035] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:00.244367 12367 consensus_queue.cc:1035] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:00.264081 11079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:00.264711 11079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:00.268976 11079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1139fe2a7ab541548c18cd9cbf968566: Sending DeleteTablet for 3 replicas of tablet 6e520431e38841fca43dd5cb7083712d
I20260925 22:50:00.269817 11985 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e520431e38841fca43dd5cb7083712d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:00 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:51610
I20260925 22:50:00.269984 12259 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e520431e38841fca43dd5cb7083712d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:00 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:36198
I20260925 22:50:00.269985 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e520431e38841fca43dd5cb7083712d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:00 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:39054
I20260925 22:50:00.270462 12383 tablet_replica.cc:330] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: stopping tablet replica
I20260925 22:50:00.270864 12383 raft_consensus.cc:2229] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:00.271196 12383 raft_consensus.cc:2258] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:00.271555 12382 tablet_replica.cc:330] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: stopping tablet replica
I20260925 22:50:00.271852 12382 raft_consensus.cc:2229] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:00.272182 12382 raft_consensus.cc:2258] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:00.273512 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11048
I20260925 22:50:00.273598 12383 ts_tablet_manager.cc:1905] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:00.274031 12381 tablet_replica.cc:330] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: stopping tablet replica
I20260925 22:50:00.274307 12381 raft_consensus.cc:2229] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:00.274530 12382 ts_tablet_manager.cc:1905] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:00.274586 12381 raft_consensus.cc:2258] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:00.275951 12381 ts_tablet_manager.cc:1905] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:00.277215 12382 ts_tablet_manager.cc:1918] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:00.277310 12382 log.cc:1196] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/6e520431e38841fca43dd5cb7083712d
I20260925 22:50:00.277365 12383 ts_tablet_manager.cc:1918] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:00.277469 12383 log.cc:1196] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/6e520431e38841fca43dd5cb7083712d
I20260925 22:50:00.277673 12382 ts_tablet_manager.cc:1939] T 6e520431e38841fca43dd5cb7083712d P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting consensus metadata
I20260925 22:50:00.277794 12383 ts_tablet_manager.cc:1939] T 6e520431e38841fca43dd5cb7083712d P aac412a347dc4554a9fbbaec501a734c: Deleting consensus metadata
I20260925 22:50:00.278558 12381 ts_tablet_manager.cc:1918] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:00.278661 12381 log.cc:1196] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/6e520431e38841fca43dd5cb7083712d
I20260925 22:50:00.278967 12381 ts_tablet_manager.cc:1939] T 6e520431e38841fca43dd5cb7083712d P 3938514dc0324358adff8e83616fcebf: Deleting consensus metadata
W20260925 22:50:00.281972 12214 connection.cc:535] server connection from 127.0.0.1:36198 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:50:00.282498 12077 connection.cc:535] server connection from 127.0.0.1:39054 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 22:50:00.282851 11941 connection.cc:535] server connection from 127.0.0.1:51610 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:50:00.292663 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.271869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.4.219.189:37741)
0925 22:50:00.272159 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.272164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.275446 (+ 3282us) server_negotiation.cc:407] Connection header received
0925 22:50:00.275497 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.275500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.275552 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.275634 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.284092 (+ 8458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.284852 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.286008 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.286374 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.286479 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.288693 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.288697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.288700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.288729 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.290415 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.291221 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.291228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.291231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.291327 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.291665 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.291668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.291672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.291915 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.292059 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.292320 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.292453 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260925 22:50:00.295702 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.285637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.4.219.189:37741)
0925 22:50:00.285756 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.285760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.286693 (+ 933us) server_negotiation.cc:407] Connection header received
0925 22:50:00.286812 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.286815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.286857 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.286933 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.287913 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.288742 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.289956 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.290312 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.290418 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.291912 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.291916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.291919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.291943 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.293713 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.294391 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.294395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.294397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.294487 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.294781 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.294785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.294787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.295077 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.295220 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.295437 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.295548 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:50:00.301810 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.272383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.4.219.188:37407)
0925 22:50:00.272498 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.272503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.292790 (+ 20287us) server_negotiation.cc:407] Connection header received
0925 22:50:00.293019 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.293023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.293091 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.293154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.293981 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.294822 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.295940 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.296256 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.296333 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.298183 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.298187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.298189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.298216 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.299881 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.300360 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.300363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.300365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.300450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.300766 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.300769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.300771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.300992 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.301120 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.301453 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.301592 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:50:00.304654 12055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:00.286035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.4.219.188:37407)
0925 22:50:00.286156 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:00.286160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:00.296146 (+ 9986us) server_negotiation.cc:407] Connection header received
0925 22:50:00.296325 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:00.296328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:00.296374 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:00.296423 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:00.297385 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.298199 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.299208 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:00.299446 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:00.299504 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:00.301260 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:00.301263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:00.301266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:00.301289 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:00.302954 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.303398 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.303402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.303403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.303489 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:00.303748 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:00.303752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:00.303754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:00.303919 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:00.304071 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:00.304300 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:50:00.304409 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:50:01.259094 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.258596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:49647)
0925 22:50:01.258884 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.258980 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
W20260925 22:50:01.259567 12052 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:42701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.259891 12393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.259488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:38283)
0925 22:50:01.259731 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.259813 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260925 22:50:01.260073 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.259717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:54229)
0925 22:50:01.259911 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.259988 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260925 22:50:01.260255 12189 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:42701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
W20260925 22:50:01.260438 12326 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:42701 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.770061 12395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1139fe2a7ab541548c18cd9cbf968566)
I20260925 22:50:01.770284 12395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } }
I20260925 22:50:01.771397 12395 leader_election.cc:288] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701), 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189:37741)
I20260925 22:50:01.771932 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.771512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52068)
0925 22:50:01.771748 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.771826 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
W20260925 22:50:01.772104 11857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.772094 12398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1139fe2a7ab541548c18cd9cbf968566)
I20260925 22:50:01.772293 12398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } }
I20260925 22:50:01.772924 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.772686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52070)
0925 22:50:01.772815 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.772864 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 22:50:01.773072 11857 leader_election.cc:334] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701): Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.773428 12398 leader_election.cc:288] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701), 2696a8da053046319f8fa2df95694dc9 (127.4.219.188:37407)
I20260925 22:50:01.773917 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.773521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52084)
0925 22:50:01.773761 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.773831 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
W20260925 22:50:01.774099 11668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.775012 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.774773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52090)
0925 22:50:01.774895 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.774938 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 22:50:01.775173 11668 leader_election.cc:334] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701): Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.776173 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.771807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.4.219.189:37741)
0925 22:50:01.772491 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.772495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.772511 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:01.772582 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.772585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.772637 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.772734 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.772966 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.773730 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.775048 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.775656 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.775772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.775830 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.775995 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.776064 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":610,"thread_start_us":54,"threads_started":1}
I20260925 22:50:01.776185 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.771653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:37741 (local address 127.0.0.1:53530)
0925 22:50:01.771968 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.771982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.772094 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.772743 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.772746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.772762 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.772865 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.772870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.773850 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.773854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.774932 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.774940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.775770 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.775774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.775862 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.775885 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.775978 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.776050 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20260925 22:50:01.776731 11693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2696a8da053046319f8fa2df95694dc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91c8fb921d25467aa14ceeabb17b6338" is_pre_election: true
I20260925 22:50:01.776942 11693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2696a8da053046319f8fa2df95694dc9 in term 1.
I20260925 22:50:01.777281 11859 leader_election.cc:302] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2696a8da053046319f8fa2df95694dc9, 91c8fb921d25467aa14ceeabb17b6338; no voters: 1139fe2a7ab541548c18cd9cbf968566
I20260925 22:50:01.777501 12395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:50:01.777520 12401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.773513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37407 (local address 127.0.0.1:36002)
0925 22:50:01.773676 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.773692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.773777 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.774311 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.774314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.774335 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.774419 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.774424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.775379 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.775385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.776215 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.776245 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.777114 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.777118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.777208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.777229 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.777319 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.777389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":17,"thread_start_us":55,"threads_started":1}
I20260925 22:50:01.777509 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.773830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.4.219.188:37407)
0925 22:50:01.774063 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.774072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.774088 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:01.774130 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.774132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.774177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.774293 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.774554 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.775285 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.776389 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.776997 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.777116 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.777169 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.777333 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.777396 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260925 22:50:01.777562 12395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1139fe2a7ab541548c18cd9cbf968566)
I20260925 22:50:01.777602 12395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:50:01.778591 12395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } }
I20260925 22:50:01.778978 12395 leader_election.cc:288] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 election: Requested vote from peers 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701), 91c8fb921d25467aa14ceeabb17b6338 (127.4.219.189:37741)
I20260925 22:50:01.779338 11693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2696a8da053046319f8fa2df95694dc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91c8fb921d25467aa14ceeabb17b6338"
I20260925 22:50:01.779448 11693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:50:01.779420 11882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91c8fb921d25467aa14ceeabb17b6338" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2696a8da053046319f8fa2df95694dc9" is_pre_election: true
I20260925 22:50:01.779642 11882 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 91c8fb921d25467aa14ceeabb17b6338 in current term 2: Already voted for candidate 2696a8da053046319f8fa2df95694dc9 in this term.
I20260925 22:50:01.780027 11667 leader_election.cc:302] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91c8fb921d25467aa14ceeabb17b6338; no voters: 1139fe2a7ab541548c18cd9cbf968566, 2696a8da053046319f8fa2df95694dc9
I20260925 22:50:01.780675 11693 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2696a8da053046319f8fa2df95694dc9 in term 2.
I20260925 22:50:01.780843 12398 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:50:01.780958 11859 leader_election.cc:302] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2696a8da053046319f8fa2df95694dc9, 91c8fb921d25467aa14ceeabb17b6338; no voters:
I20260925 22:50:01.781075 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.779073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52106)
0925 22:50:01.779222 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.781016 (+ 1794us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 22:50:01.781111 12395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:50:01.781316 12395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [term 2 LEADER]: Becoming Leader. State: Replica: 2696a8da053046319f8fa2df95694dc9, State: Running, Role: LEADER
I20260925 22:50:01.781663 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.781501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52122)
0925 22:50:01.781576 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.781618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 22:50:01.781607 12395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } }
W20260925 22:50:01.781865 11857 leader_election.cc:334] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701): Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
I20260925 22:50:01.782672 12405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2696a8da053046319f8fa2df95694dc9. Latest consensus state: current_term: 2 leader_uuid: "2696a8da053046319f8fa2df95694dc9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:01.782793 12405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:01.783090 12407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:01.785454 12407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:01.785823 12407 catalog_manager.cc:1259] Loaded cluster ID: 71ed8328425d4a69af9fe177889797a5
I20260925 22:50:01.785866 12407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:01.786382 12407 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:01.786755 12407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Loaded TSK: 0
I20260925 22:50:01.787390 12407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:01.821158 11870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:01.824606 11870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:01.827378 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.827149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52138)
0925 22:50:01.827263 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.827314 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:50:01.827425 11693 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [term 2 FOLLOWER]: Refusing update from remote peer 2696a8da053046319f8fa2df95694dc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 22:50:01.827955 12395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 22:50:01.828038 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.827841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.0.0.1:52150)
0925 22:50:01.827934 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.827980 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:50:01.828156 11857 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 -> Peer 1139fe2a7ab541548c18cd9cbf968566 (127.4.219.190:42701): Couldn't send request to peer 1139fe2a7ab541548c18cd9cbf968566. Status: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:50:01.829470 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2696a8da053046319f8fa2df95694dc9. Latest consensus state: current_term: 2 leader_uuid: "2696a8da053046319f8fa2df95694dc9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:01.829632 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:01.830619 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2696a8da053046319f8fa2df95694dc9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:01.830729 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:01.831019 12398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2696a8da053046319f8fa2df95694dc9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1139fe2a7ab541548c18cd9cbf968566" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 42701 } } peers { permanent_uuid: "91c8fb921d25467aa14ceeabb17b6338" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 37741 } attrs { promote: false } } peers { permanent_uuid: "2696a8da053046319f8fa2df95694dc9" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37407 } attrs { promote: false } } }
I20260925 22:50:01.831131 12398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91c8fb921d25467aa14ceeabb17b6338 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:50:01.840430 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.836109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44083 (local address 127.0.0.1:39064)
0925 22:50:01.836252 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.836266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.836344 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.836756 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.836759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.836778 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.836887 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.836893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.837907 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.837910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.839034 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.839043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.840045 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.840048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.840130 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.840152 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.840218 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.840290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:50:01.840866 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.836227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.4.219.130:44083)
0925 22:50:01.836465 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.836469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.836483 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:01.836571 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.836574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.836642 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.836735 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.837016 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.837794 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.839168 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.839795 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.839903 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.839991 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.840648 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.840728 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260925 22:50:01.841400 12122 tablet_service.cc:1467] Processing CreateTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 (DEFAULT_TABLE table=test-table [id=8fdd0de975be4290ba58891d143f996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:01.841692 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d53369ac404d5a950bbc74d61fcd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:01.841857 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.837000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.4.219.131:46403)
0925 22:50:01.837965 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.837971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.837987 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:01.838044 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.838047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.838100 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.838208 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.838471 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.839016 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.840142 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.841297 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.841430 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.841495 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.841646 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.841724 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":77,"threads_started":1}
I20260925 22:50:01.841858 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.836766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:46403 (local address 127.0.0.1:36212)
0925 22:50:01.837549 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.837562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.837651 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.838227 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.838230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.838249 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.838342 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.838348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.839135 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.839139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.839978 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.839986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.841428 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.841431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.841527 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.841554 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.841631 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.841713 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":530,"thread_start_us":114,"threads_started":1}
I20260925 22:50:01.842384 12259 tablet_service.cc:1467] Processing CreateTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 (DEFAULT_TABLE table=test-table [id=8fdd0de975be4290ba58891d143f996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:01.842664 12259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d53369ac404d5a950bbc74d61fcd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:01.844098 12418 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: Bootstrap starting.
I20260925 22:50:01.844504 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.836762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41391 (local address 127.0.0.1:51616)
0925 22:50:01.837023 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.837038 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.837126 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.837509 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.837512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.837528 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.837610 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.837615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.838712 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.838716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.839857 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.839869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.844092 (+ 4223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.844097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.844207 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.844234 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.844312 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.844386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260925 22:50:01.844941 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.837029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.4.219.129:41391)
0925 22:50:01.837279 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.837284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.837298 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:01.837347 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.837350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.837392 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.837700 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.837801 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.838587 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.840015 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.840866 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.841199 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.841264 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.844715 (+ 3451us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.844818 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260925 22:50:01.845317 12419 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Bootstrap starting.
I20260925 22:50:01.845470 11985 tablet_service.cc:1467] Processing CreateTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 (DEFAULT_TABLE table=test-table [id=8fdd0de975be4290ba58891d143f996f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:01.845640 12418 tablet_bootstrap.cc:654] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:01.845767 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d53369ac404d5a950bbc74d61fcd97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:01.846159 12419 tablet_bootstrap.cc:654] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:01.847200 12418 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: No bootstrap required, opened a new log
I20260925 22:50:01.847301 12418 ts_tablet_manager.cc:1397] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:01.847342 12419 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: No bootstrap required, opened a new log
I20260925 22:50:01.847429 12419 ts_tablet_manager.cc:1397] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:01.847832 12419 raft_consensus.cc:348] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.847833 12418 raft_consensus.cc:348] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.847968 12419 raft_consensus.cc:374] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:01.847980 12418 raft_consensus.cc:374] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:01.848014 12419 raft_consensus.cc:729] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc9eebbb4ea4e03beb418e9d4f09c0a, State: Initialized, Role: FOLLOWER
I20260925 22:50:01.848021 12418 raft_consensus.cc:729] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aac412a347dc4554a9fbbaec501a734c, State: Initialized, Role: FOLLOWER
I20260925 22:50:01.848155 12418 consensus_queue.cc:260] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.848155 12419 consensus_queue.cc:260] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.848500 12419 ts_tablet_manager.cc:1428] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:01.848506 12418 ts_tablet_manager.cc:1428] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:50:01.849762 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.849303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:45465)
0925 22:50:01.849580 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.849648 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":130,"threads_started":1}
I20260925 22:50:01.850589 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.850311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:55401)
0925 22:50:01.850459 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.850515 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 22:50:01.851190 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.849622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:60591)
0925 22:50:01.851029 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.851091 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":95,"threads_started":1}
I20260925 22:50:01.852258 12325 heartbeater.cc:503] Master 127.4.219.188:37407 was elected leader, sending a full tablet report...
I20260925 22:50:01.852478 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.851707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:53597)
0925 22:50:01.852308 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.852366 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
I20260925 22:50:01.852595 12422 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: Bootstrap starting.
I20260925 22:50:01.852741 12188 heartbeater.cc:503] Master 127.4.219.188:37407 was elected leader, sending a full tablet report...
I20260925 22:50:01.853776 12422 tablet_bootstrap.cc:654] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:01.855008 12422 tablet_bootstrap.cc:492] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: No bootstrap required, opened a new log
I20260925 22:50:01.855118 12422 ts_tablet_manager.cc:1397] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 22:50:01.855517 12422 raft_consensus.cc:348] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.855630 12422 raft_consensus.cc:374] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:01.855674 12422 raft_consensus.cc:729] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3938514dc0324358adff8e83616fcebf, State: Initialized, Role: FOLLOWER
I20260925 22:50:01.855818 12422 consensus_queue.cc:260] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.856113 12422 ts_tablet_manager.cc:1428] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:01.856444 12423 raft_consensus.cc:482] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:01.856573 12423 raft_consensus.cc:504] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.857092 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.856665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:40983)
0925 22:50:01.856920 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.856993 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
I20260925 22:50:01.857757 12423 leader_election.cc:288] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403), 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:50:01.858014 12051 heartbeater.cc:503] Master 127.4.219.188:37407 was elected leader, sending a full tablet report...
I20260925 22:50:01.859146 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.858876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:45327)
0925 22:50:01.859021 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.859078 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 22:50:01.862690 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.857874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:46403 (local address 127.4.219.130:43933)
0925 22:50:01.858533 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.858550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.858695 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.859130 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.859134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.859155 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.859254 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.859261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.860395 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.860398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.861682 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.861692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.862360 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.862362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.862446 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.862466 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.862537 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.862593 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577}
I20260925 22:50:01.862743 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.858145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:43933 (local address 127.4.219.131:46403)
0925 22:50:01.858281 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.858286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.858615 (+ 329us) server_negotiation.cc:407] Connection header received
0925 22:50:01.858836 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.858841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.858908 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.858964 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.859397 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.860260 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.861801 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.862249 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.862363 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.862413 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.862563 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.862629 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:50:01.863250 12279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d53369ac404d5a950bbc74d61fcd97" candidate_uuid: "aac412a347dc4554a9fbbaec501a734c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" is_pre_election: true
I20260925 22:50:01.863539 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.858527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:53147 (local address 127.4.219.129:41391)
0925 22:50:01.858814 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.858819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.859809 (+ 990us) server_negotiation.cc:407] Connection header received
0925 22:50:01.859978 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.859981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.860021 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.860066 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:01.860377 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.860923 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.862208 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.862772 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.862892 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.862937 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.863338 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.863418 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260925 22:50:01.863511 12279 raft_consensus.cc:2454] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac412a347dc4554a9fbbaec501a734c in term 0.
I20260925 22:50:01.863555 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.858103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41391 (local address 127.4.219.130:53147)
0925 22:50:01.859730 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.859744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.859869 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.860182 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:01.860185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:01.860199 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:01.860277 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.860283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.861030 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.861034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.862105 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:01.862112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.863127 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.863131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:01.863218 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.863240 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:50:01.863325 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:50:01.863408 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":113,"threads_started":1}
I20260925 22:50:01.863878 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d53369ac404d5a950bbc74d61fcd97" candidate_uuid: "aac412a347dc4554a9fbbaec501a734c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3938514dc0324358adff8e83616fcebf" is_pre_election: true
I20260925 22:50:01.863951 12078 leader_election.cc:302] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aac412a347dc4554a9fbbaec501a734c, fcc9eebbb4ea4e03beb418e9d4f09c0a; no voters:
I20260925 22:50:01.864014 12005 raft_consensus.cc:2454] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aac412a347dc4554a9fbbaec501a734c in term 0.
I20260925 22:50:01.864214 12423 raft_consensus.cc:2781] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:01.864279 12423 raft_consensus.cc:482] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:01.864315 12423 raft_consensus.cc:3037] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:01.865366 12423 raft_consensus.cc:504] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.865741 12423 leader_election.cc:288] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [CANDIDATE]: Term 1 election: Requested vote from peers fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403), 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391)
I20260925 22:50:01.866137 12279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d53369ac404d5a950bbc74d61fcd97" candidate_uuid: "aac412a347dc4554a9fbbaec501a734c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a"
I20260925 22:50:01.866241 12279 raft_consensus.cc:3037] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:01.866194 12005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d53369ac404d5a950bbc74d61fcd97" candidate_uuid: "aac412a347dc4554a9fbbaec501a734c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3938514dc0324358adff8e83616fcebf"
I20260925 22:50:01.866341 12005 raft_consensus.cc:3037] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:01.867259 12279 raft_consensus.cc:2454] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac412a347dc4554a9fbbaec501a734c in term 1.
I20260925 22:50:01.867527 12078 leader_election.cc:302] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aac412a347dc4554a9fbbaec501a734c, fcc9eebbb4ea4e03beb418e9d4f09c0a; no voters:
I20260925 22:50:01.867647 12005 raft_consensus.cc:2454] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aac412a347dc4554a9fbbaec501a734c in term 1.
I20260925 22:50:01.867702 12423 raft_consensus.cc:2781] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:01.867913 12423 raft_consensus.cc:686] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 1 LEADER]: Becoming Leader. State: Replica: aac412a347dc4554a9fbbaec501a734c, State: Running, Role: LEADER
I20260925 22:50:01.868191 12423 consensus_queue.cc:237] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } }
I20260925 22:50:01.868953 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.868686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:57525)
0925 22:50:01.868827 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.868884 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 22:50:01.870725 11872 catalog_manager.cc:5495] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c reported cstate change: term changed from 0 to 1, leader changed from <none> to aac412a347dc4554a9fbbaec501a734c (127.4.219.130). New cstate: current_term: 1 leader_uuid: "aac412a347dc4554a9fbbaec501a734c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aac412a347dc4554a9fbbaec501a734c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 } health_report { overall_health: UNKNOWN } } }
I20260925 22:50:01.883829 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.879959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.4.219.130:44083)
0925 22:50:01.880095 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:01.880100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:01.880115 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:01.880227 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:01.880230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:01.880286 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:01.880395 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:01.880689 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.881206 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.882211 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:01.882438 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:01.882563 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:01.882863 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:01.883467 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:01.883589 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:01.883630 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 22:50:01.883686 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:50:01.886745 12005 raft_consensus.cc:1261] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf [term 1 FOLLOWER]: Refusing update from remote peer aac412a347dc4554a9fbbaec501a734c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:01.886831 12279 raft_consensus.cc:1261] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Refusing update from remote peer aac412a347dc4554a9fbbaec501a734c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:01.887306 12423 consensus_queue.cc:1035] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3938514dc0324358adff8e83616fcebf" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:01.887549 12431 consensus_queue.cc:1035] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc9eebbb4ea4e03beb418e9d4f09c0a" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 46403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:01.889653 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.889316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:34927)
0925 22:50:01.889497 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.889561 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 22:50:01.892247 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.889060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:50199)
0925 22:50:01.892101 (+ 3041us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.892171 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2966}
I20260925 22:50:01.893534 12434 mvcc.cc:205] Tried to move back new op lower bound from 7333382561322995712 to 7333382561255350272. Current Snapshot: MvccSnapshot[applied={T|T < 7333382561322995712 or (T in {7333382561322995712})}]
I20260925 22:50:01.893748 12435 mvcc.cc:205] Tried to move back new op lower bound from 7333382561322995712 to 7333382561255350272. Current Snapshot: MvccSnapshot[applied={T|T < 7333382561322995712}]
I20260925 22:50:01.894150 12429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.893887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.129:56509)
0925 22:50:01.894031 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.894085 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 22:50:01.897166 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.896894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.131:59407)
0925 22:50:01.897050 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.897101 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 22:50:01.897716 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.892080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:46341)
0925 22:50:01.897591 (+ 5511us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.897644 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5452}
I20260925 22:50:01.898736 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.898506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:42701 (local address 127.4.219.130:40585)
0925 22:50:01.898619 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.898671 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:42701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:50:01.901392 11872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:01.901551 11872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:01.905172 11872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2696a8da053046319f8fa2df95694dc9: Sending DeleteTablet for 3 replicas of tablet 06d53369ac404d5a950bbc74d61fcd97
I20260925 22:50:01.905871 11985 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:01 UTC) from {username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:51616
I20260925 22:50:01.905925 12259 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:01 UTC) from {username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:36212
I20260925 22:50:01.906221 12444 tablet_replica.cc:330] T 06d53369ac404d5a950bbc74d61fcd97 P 3938514dc0324358adff8e83616fcebf: stopping tablet replica
I20260925 22:50:01.906404 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d53369ac404d5a950bbc74d61fcd97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:01 UTC) from {username='oryx', principal='oryx/127.4.219.188@KRBTEST.COM'} at 127.0.0.1:39064
I20260925 22:50:01.906690 12446 tablet_replica.cc:330] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: stopping tablet replica
I20260925 22:50:01.906708 12445 tablet_replica.cc:330] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: stopping tablet replica
I20260925 22:50:01.906816 12445 raft_consensus.cc:2229] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:01.906847 12446 raft_consensus.cc:2229] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:01.906950 12445 raft_consensus.cc:2258] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:01.907043 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11827
I20260925 22:50:01.907296 12446 raft_consensus.cc:2258] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:01.908391 12445 ts_tablet_manager.cc:1905] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:01.911777 12445 ts_tablet_manager.cc:1918] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:01.911860 12445 log.cc:1196] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/06d53369ac404d5a950bbc74d61fcd97
I20260925 22:50:01.912124 12445 ts_tablet_manager.cc:1939] T 06d53369ac404d5a950bbc74d61fcd97 P fcc9eebbb4ea4e03beb418e9d4f09c0a: Deleting consensus metadata
I20260925 22:50:01.913797 11859 catalog_manager.cc:4841] TS fcc9eebbb4ea4e03beb418e9d4f09c0a (127.4.219.131:46403): tablet 06d53369ac404d5a950bbc74d61fcd97 (table test-table [id=8fdd0de975be4290ba58891d143f996f]) successfully deleted
I20260925 22:50:01.914474 12446 ts_tablet_manager.cc:1905] T 06d53369ac404d5a950bbc74d61fcd97 P aac412a347dc4554a9fbbaec501a734c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:50:01.915225 12396 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:01.914727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41391 (local address 127.0.0.1:51624)
0925 22:50:01.914846 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.914862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:01.914946 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:01.915096 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:41391: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 22:50:01.915352 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12059
W20260925 22:50:01.915371 11858 catalog_manager.cc:4584] TS 3938514dc0324358adff8e83616fcebf (127.4.219.129:41391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06d53369ac404d5a950bbc74d61fcd97: Network error: Client connection negotiation failed: client connection to 127.4.219.129:41391: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:50:01.920979 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12195
I20260925 22:50:01.921134 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:01.920860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44083 (local address 127.0.0.1:39084)
0925 22:50:01.920988 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:01.921043 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.130:44083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 22:50:01.928123 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11544
I20260925 22:50:01.933787 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 11655
2026-09-25T22:50:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5307 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:01 dist-test-slave-kb1g krb5kdc[12449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:01 dist-test-slave-kb1g krb5kdc[12449](info): set up 2 sockets
Sep 25 22:50:01 dist-test-slave-kb1g krb5kdc[12449](info): commencing operation
krb5kdc: starting...
W20260925 22:50:02.977142 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab.
Entry for principal oryx/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab.
I20260925 22:50:03.099556 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45645
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab
--principal=oryx/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:45645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:03.207860 12462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:03.208112 12462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:03.208150 12462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:03.212055 12462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:03.212112 12462 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:50:03.212129 12462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:03.212147 12462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:03.212163 12462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:03.212178 12462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:03.217442 12462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:45645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.190.keytab
--principal=oryx/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:03.218467 12462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:03.219430 12462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:03.227474 12467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.227521 12468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.227531 12470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.227654 12462 server_base.cc:1034] running on GCE node
I20260925 22:50:03.228305 12462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:03.229418 12462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:03.230623 12462 hybrid_clock.cc:648] HybridClock initialized: now 1790376603230610 us; error 27 us; skew 500 ppm
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:03.233428 12462 init.cc:377] Logged in from keytab as oryx/127.4.219.190@KRBTEST.COM (short username oryx)
I20260925 22:50:03.234231 12462 webserver.cc:460] Webserver started at http://127.4.219.190:33347/ using document root <none> and password file <none>
I20260925 22:50:03.234732 12462 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:03.234787 12462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:03.234946 12462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:03.236986 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "b24cd0a84a8eb2c8b9316a7cd3d082dc"
server_key_iv: "9743fe320d41a3f7bbaf05af6b6d40ea"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.237478 12462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "b24cd0a84a8eb2c8b9316a7cd3d082dc"
server_key_iv: "9743fe320d41a3f7bbaf05af6b6d40ea"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.241189 12462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:50:03.243520 12477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.244781 12462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:03.244900 12462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "b24cd0a84a8eb2c8b9316a7cd3d082dc"
server_key_iv: "9743fe320d41a3f7bbaf05af6b6d40ea"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.245028 12462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:03.260175 12462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:03.263082 12462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:03.263276 12462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:03.270977 12462 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:45645
I20260925 22:50:03.270994 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:45645 every 8 connection(s)
I20260925 22:50:03.271906 12462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:03.274639 12540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:03.275219 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12462
I20260925 22:50:03.275341 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:50:03.275642 4974 external_mini_cluster.cc:1428] Setting key 9866fa8260a498e2931b4056f9faa8f6
I20260925 22:50:03.281445 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Bootstrap starting.
I20260925 22:50:03.283875 12540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Neither blocks nor log segments found. Creating new log.
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:03.284672 12540 log.cc:824] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:03.286577 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: No bootstrap required, opened a new log
I20260925 22:50:03.289659 12540 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } }
I20260925 22:50:03.289891 12540 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:03.289947 12540 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5e8c9ebf264b09b5cc2c3b158d5a75, State: Initialized, Role: FOLLOWER
I20260925 22:50:03.290370 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.277111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.4.219.190:45645)
0925 22:50:03.277548 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.277565 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.277600 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:50:03.278258 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.278276 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.278627 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.278978 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.279798 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.280875 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.281810 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.282300 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.282483 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.285075 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.285080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.285095 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:03.285129 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.287380 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.287845 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.287850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.287855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.287968 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.288253 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.288256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.288258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.288722 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.288943 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.289225 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.289530 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":129,"threads_started":1}
I20260925 22:50:03.290421 12540 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } }
I20260925 22:50:03.290582 12540 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:03.290644 12540 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:03.290760 12540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:03.291682 12540 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } }
I20260925 22:50:03.292023 12540 leader_election.cc:302] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc5e8c9ebf264b09b5cc2c3b158d5a75; no voters:
I20260925 22:50:03.292352 12540 leader_election.cc:288] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:03.292431 12545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:03.292656 12545 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [term 1 LEADER]: Becoming Leader. State: Replica: dc5e8c9ebf264b09b5cc2c3b158d5a75, State: Running, Role: LEADER
I20260925 22:50:03.292971 12545 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } }
I20260925 22:50:03.293994 12540 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:03.294739 12546 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc5e8c9ebf264b09b5cc2c3b158d5a75. Latest consensus state: current_term: 1 leader_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } } }
I20260925 22:50:03.294862 12546 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:03.295166 12547 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc5e8c9ebf264b09b5cc2c3b158d5a75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45645 } } }
I20260925 22:50:03.295264 12547 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:03.296036 12550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:03.298693 12550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:03.304052 12550 catalog_manager.cc:1347] Generated new cluster ID: d9e1eab8dcc046bc808fb385c031a35c
I20260925 22:50:03.304123 12550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:03.328425 12550 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:03.329483 12550 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:03.349622 12550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Generated new TSK 0
I20260925 22:50:03.350543 12550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
I20260925 22:50:03.389428 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:03.503201 12566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:03.503468 12566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:03.503513 12566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:03.507576 12566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:03.507647 12566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:03.507686 12566 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:50:03.507719 12566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:03.513330 12566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:03.514398 12566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:03.515482 12566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:03.525441 12572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.525518 12566 server_base.cc:1034] running on GCE node
W20260925 22:50:03.525475 12574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.525465 12571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.526046 12566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:03.526684 12566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:03.527853 12566 hybrid_clock.cc:648] HybridClock initialized: now 1790376603527826 us; error 42 us; skew 500 ppm
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:03.530807 12566 init.cc:377] Logged in from keytab as oryx/127.4.219.129@KRBTEST.COM (short username oryx)
I20260925 22:50:03.531746 12566 webserver.cc:460] Webserver started at http://127.4.219.129:39879/ using document root <none> and password file <none>
I20260925 22:50:03.532311 12566 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:03.532374 12566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:03.532653 12566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:03.534667 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "b14bd284fb094d899426a409f1a4a42e"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "3d9a3391be47aac108bf5cd9b095f328"
server_key_iv: "6b6f85fae3d4dfb5cfc5c9b6a6c8df1a"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.535254 12566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "b14bd284fb094d899426a409f1a4a42e"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "3d9a3391be47aac108bf5cd9b095f328"
server_key_iv: "6b6f85fae3d4dfb5cfc5c9b6a6c8df1a"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.539227 12566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:50:03.541709 12581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.543099 12566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:03.543200 12566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "b14bd284fb094d899426a409f1a4a42e"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "3d9a3391be47aac108bf5cd9b095f328"
server_key_iv: "6b6f85fae3d4dfb5cfc5c9b6a6c8df1a"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.543371 12566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:03.558267 12566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:03.561574 12566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:03.561774 12566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:03.563238 12566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:03.563285 12566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.563324 12566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:03.563344 12566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:03.574057 12566 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:43273
I20260925 22:50:03.574121 12696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:43273 every 8 connection(s)
I20260925 22:50:03.574997 12566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:03.575867 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12566
I20260925 22:50:03.576006 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:50:03.576366 4974 external_mini_cluster.cc:1428] Setting key 17b019bb946d80eb229576f39abfd902
I20260925 22:50:03.577318 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.564085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52817 (local address 127.4.219.190:45645)
0925 22:50:03.564254 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.564259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.565151 (+ 892us) server_negotiation.cc:407] Connection header received
0925 22:50:03.566316 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.566323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.566398 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.566507 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.567852 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.568388 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.569702 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.569948 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.570061 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.573289 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.573293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.573296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:03.573325 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.574931 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.575501 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.575504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.575506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.575596 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.576052 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.576056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.576058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.576303 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.576414 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.577030 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.577171 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:50:03.578315 12590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.564480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.129:52817)
0925 22:50:03.564980 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:03.565030 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:03.566092 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:03.566701 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:03.566712 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:03.567067 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:03.567683 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.567701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.568521 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.568524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.569565 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.569575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.570112 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.570116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.570320 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.570918 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:03.570943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:03.573109 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:03.575050 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.575056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.575071 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.575345 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.575709 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.575713 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.575719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.575923 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.576480 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:03.576487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:03.576776 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 22:50:03.576968 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 22:50:03.577274 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260925 22:50:03.579787 12697 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45645
I20260925 22:50:03.579945 12697 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:03.580551 12697 heartbeater.cc:511] Master 127.4.219.190:45645 requested a full tablet report, sending...
I20260925 22:50:03.582046 12493 ts_manager.cc:194] Registered new tserver with Master: b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273)
I20260925 22:50:03.583247 12493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:52817
WARNING: no policy specified for oryx/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.130@KRBTEST.COM" created.
I20260925 22:50:03.593027 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.582140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:58207 (local address 127.4.219.190:45645)
0925 22:50:03.582248 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.582252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.586317 (+ 4065us) server_negotiation.cc:407] Connection header received
0925 22:50:03.586468 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.586471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.586514 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.586589 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.587068 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.587580 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.588413 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.588713 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.588829 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.589950 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.589953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.589956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:03.589979 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.591405 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.591837 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.591841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.591843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.591922 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.592182 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.592186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.592188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.592363 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.592463 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.592757 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.592862 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:50:03.593060 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.582065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.129:58207)
0925 22:50:03.586223 (+ 4158us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:03.586241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:03.586348 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:03.586615 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:03.586617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:03.586812 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:03.586965 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.586972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.587691 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.587694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.588284 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.588293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.588834 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.588838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.588931 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.589472 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:03.589491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:03.589810 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:03.591538 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.591542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.591544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.591743 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.592010 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.592013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.592015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.592098 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.592456 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:03.592459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:03.592585 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 22:50:03.592704 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:50:03.592852 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4040,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
I20260925 22:50:03.603621 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:03.713395 12706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:03.713649 12706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:03.713693 12706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:03.717602 12706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:03.717657 12706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:03.717696 12706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:50:03.717728 12706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:03.722910 12706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:03.723851 12706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:03.724800 12706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:03.733979 12711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.734047 12714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.734153 12706 server_base.cc:1034] running on GCE node
W20260925 22:50:03.734053 12712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.734570 12706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:03.735188 12706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:03.736369 12706 hybrid_clock.cc:648] HybridClock initialized: now 1790376603736349 us; error 30 us; skew 500 ppm
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:03.739246 12706 init.cc:377] Logged in from keytab as oryx/127.4.219.130@KRBTEST.COM (short username oryx)
I20260925 22:50:03.740123 12706 webserver.cc:460] Webserver started at http://127.4.219.130:36843/ using document root <none> and password file <none>
I20260925 22:50:03.740726 12706 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:03.740808 12706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:03.741031 12706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:03.743077 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "84d43498a18e240ca8e0ef249c72c0cb"
server_key_iv: "01f66c5b181498f5865790addb2c0dfa"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.743647 12706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "84d43498a18e240ca8e0ef249c72c0cb"
server_key_iv: "01f66c5b181498f5865790addb2c0dfa"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.747370 12706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:50:03.749822 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.751006 12706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:03.751129 12706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "84d43498a18e240ca8e0ef249c72c0cb"
server_key_iv: "01f66c5b181498f5865790addb2c0dfa"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.751235 12706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:03.770845 12706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:03.774137 12706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:03.774335 12706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:03.775795 12706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:03.775870 12706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.775929 12706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:03.775962 12706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:03.786909 12706 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44767
I20260925 22:50:03.786954 12836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44767 every 8 connection(s)
I20260925 22:50:03.787997 12706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:50:03.789955 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12706
I20260925 22:50:03.789978 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.776742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:35851 (local address 127.4.219.190:45645)
0925 22:50:03.777057 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.777063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.777895 (+ 832us) server_negotiation.cc:407] Connection header received
0925 22:50:03.779191 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.779196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.779259 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.779368 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.780775 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.781386 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.782634 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.782904 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.782969 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.785985 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.785988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.785990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:03.786016 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.787699 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.788301 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.788308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.788313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.788426 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.788857 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.788862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.788866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.789094 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.789173 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.789626 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.789799 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260925 22:50:03.790072 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:50:03.790342 4974 external_mini_cluster.cc:1428] Setting key aefe1eb28ba40e2682cac50eb658eae1
I20260925 22:50:03.790802 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.777233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.130:35851)
0925 22:50:03.777711 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:03.777771 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:03.778951 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:03.779574 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:03.779583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:03.780021 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:03.780637 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.780653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.781513 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.781516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.782475 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.782484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.783142 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.783145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.783282 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.784215 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:03.784231 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:03.785801 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:03.787810 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.787814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.787826 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.788185 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.788561 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.788564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.788566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.788736 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.789168 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:03.789173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:03.789389 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 22:50:03.789531 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:50:03.789822 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260925 22:50:03.792061 12837 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45645
I20260925 22:50:03.792203 12837 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:03.792790 12837 heartbeater.cc:511] Master 127.4.219.190:45645 requested a full tablet report, sending...
I20260925 22:50:03.793813 12494 ts_manager.cc:194] Registered new tserver with Master: 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:03.794590 12494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:35851
WARNING: no policy specified for oryx/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.131@KRBTEST.COM" created.
I20260925 22:50:03.803735 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.794190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38763 (local address 127.4.219.190:45645)
0925 22:50:03.794330 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.794334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.794538 (+ 204us) server_negotiation.cc:407] Connection header received
0925 22:50:03.794669 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.794673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.794718 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.794805 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.796769 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.797634 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.798990 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.799339 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.799434 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.800688 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:03.800691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:03.800693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:03.800713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:03.802182 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.802557 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.802560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.802561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.802632 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:03.802928 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:03.802930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:03.802931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:03.803122 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:03.803197 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:03.803480 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 22:50:03.803594 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:50:03.803830 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.794150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.130:38763)
0925 22:50:03.794458 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:03.794469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:03.794581 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:03.794819 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:03.794821 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:03.795043 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:03.796616 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.796631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.797809 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.797815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.798862 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.798874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.799474 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.799480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.799600 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.800077 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:03.800096 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:03.800555 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:03.802271 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.802273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.802275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.802472 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.802724 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:03.802730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:03.802735 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:03.802845 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:03.803196 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:03.803199 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:03.803288 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 22:50:03.803418 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:50:03.803608 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
I20260925 22:50:03.819788 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:03.928778 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:03.929059 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:03.929126 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:03.933327 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:03.933401 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:03.933465 12846 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:50:03.933542 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:03.938612 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.219.190:45645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:03.939646 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:03.940725 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:03.950315 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:03.950352 12851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.950397 12846 server_base.cc:1034] running on GCE node
W20260925 22:50:03.950330 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:03.951077 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:03.951736 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:03.952943 12846 hybrid_clock.cc:648] HybridClock initialized: now 1790376603952922 us; error 33 us; skew 500 ppm
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:03.955790 12846 init.cc:377] Logged in from keytab as oryx/127.4.219.131@KRBTEST.COM (short username oryx)
I20260925 22:50:03.956763 12846 webserver.cc:460] Webserver started at http://127.4.219.131:38917/ using document root <none> and password file <none>
I20260925 22:50:03.957334 12846 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:03.957432 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:03.957650 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:03.959648 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "f712049b96fa4cb9b47274f558d2ec70"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "bc4b01a884c25798b03eb6c0b8d12b68"
server_key_iv: "a8d6a1c32685380bc97d0ff6a5a5ff54"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.960204 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "f712049b96fa4cb9b47274f558d2ec70"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "bc4b01a884c25798b03eb6c0b8d12b68"
server_key_iv: "a8d6a1c32685380bc97d0ff6a5a5ff54"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.963913 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:50:03.966322 12861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.967525 12846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:03.967645 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "f712049b96fa4cb9b47274f558d2ec70"
format_stamp: "Formatted at 2026-09-25 22:50:03 on dist-test-slave-kb1g"
server_key: "bc4b01a884c25798b03eb6c0b8d12b68"
server_key_iv: "a8d6a1c32685380bc97d0ff6a5a5ff54"
server_key_version: "encryptionkey@0"
I20260925 22:50:03.967748 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:03.985256 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:03.988194 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:03.988391 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:03.989821 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:03.989893 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:03.989961 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:03.990007 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:50:03 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:04.000931 12846 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:41149
I20260925 22:50:04.000980 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:41149 every 8 connection(s)
I20260925 22:50:04.002065 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:50:04.003787 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.990663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:42203 (local address 127.4.219.190:45645)
0925 22:50:03.990803 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:03.990808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:03.991700 (+ 892us) server_negotiation.cc:407] Connection header received
0925 22:50:03.992822 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:03.992826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:03.992890 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:03.992957 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:03.994544 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.995302 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.996999 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.997324 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.997454 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.000062 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:04.000067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:04.000070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:04.000099 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:04.001821 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.002367 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.002374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.002379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.002480 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.002888 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.002891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.002892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.003075 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:04.003151 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.003581 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.003670 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:50:04.004629 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:03.991084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.131:42203)
0925 22:50:03.991527 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:03.991577 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:03.992573 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:03.993342 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:03.993351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:03.993788 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:03.994399 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.994415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.995424 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.995427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.996866 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:03.996876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:03.997492 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:03.997496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:03.997668 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:03.998270 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:03.998286 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:03.999873 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:04.001926 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.001931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.001943 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.002268 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.002593 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.002595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.002598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.002767 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.003165 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:04.003170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:04.003376 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.003538 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.003770 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260925 22:50:04.005705 12977 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45645
I20260925 22:50:04.005862 12977 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:04.005861 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 12846
I20260925 22:50:04.005982 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:50:04.006316 4974 external_mini_cluster.cc:1428] Setting key 96612b82aee87db29a149cea92fb0142
I20260925 22:50:04.006379 12977 heartbeater.cc:511] Master 127.4.219.190:45645 requested a full tablet report, sending...
I20260925 22:50:04.007378 12494 ts_manager.cc:194] Registered new tserver with Master: f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149)
I20260925 22:50:04.007927 12494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:42203
I20260925 22:50:04.008317 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:50:04.018069 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.010486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.4.219.131:33423)
0925 22:50:04.010731 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.010744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.010825 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.011064 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.011066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.011224 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:04.011317 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.011322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.012369 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.012371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.013040 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.013048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.013662 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.013665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.013757 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.014291 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:04.014311 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:04.014773 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:04.016656 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.016658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.016660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.016838 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.017127 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.017129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.017130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.017201 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.017651 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:04.017653 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:04.017731 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.017813 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.017910 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260925 22:50:04.018236 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.008277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:33423 (local address 127.4.219.190:45645)
0925 22:50:04.008401 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.008405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.010816 (+ 2411us) server_negotiation.cc:407] Connection header received
0925 22:50:04.010894 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.010897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.010948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.011040 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.011420 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.012244 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.013157 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.013534 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.013659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.014914 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:04.014917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:04.014919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:04.014939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:04.016567 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.016930 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.016937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.016942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.017035 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.017292 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.017296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.017298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.017556 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:04.017664 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.017894 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.018028 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:50:04.032840 12543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:04.027029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.4.219.190:45645)
0925 22:50:04.027141 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.027144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.027154 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:50:04.027184 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.027186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.027219 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.027301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.027944 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.028431 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.029280 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.029496 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.029598 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.032668 (+ 3070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42394: BlockingRecv error: recv got EOF from 127.0.0.1:42394 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:04.041352 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.034037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.4.219.190:45645)
0925 22:50:04.034158 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.034162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.034320 (+ 158us) server_negotiation.cc:407] Connection header received
0925 22:50:04.034408 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.034410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.034452 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.034516 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.035024 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.035494 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.036197 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.036388 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.036468 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.038507 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:04.038510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:04.038512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:04.038533 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:04.039911 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.040272 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.040275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.040276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.040344 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.040645 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.040647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.040648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.040803 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:04.040910 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.041103 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.041187 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:50:04.043905 12494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:04.046954 12494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:04.061655 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.057029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.4.219.129:43273)
0925 22:50:04.057365 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.057378 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.057397 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:50:04.057465 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.057470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.057628 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.057765 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.058080 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059157 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.060332 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.060980 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.061085 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.061242 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.061404 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.061505 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
I20260925 22:50:04.061661 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.056818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:43273 (local address 127.0.0.1:33122)
0925 22:50:04.057098 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.057110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.057182 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.057757 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.057759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.057791 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.057956 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.057962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059283 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.059286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.060210 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.060220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.061095 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.061099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.061213 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.061251 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.061370 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.061492 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":62,"spinlock_wait_cycles":1280,"thread_start_us":83,"threads_started":1}
I20260925 22:50:04.062409 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.055665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41149 (local address 127.0.0.1:53812)
0925 22:50:04.056763 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.056813 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.056972 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.057569 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.057575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.057595 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.057735 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.057744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059265 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.059268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.060776 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.060785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.062009 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.062012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.062098 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.062116 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.062202 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.062271 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":185,"spinlock_wait_cycles":7040,"thread_start_us":81,"threads_started":1}
I20260925 22:50:04.062455 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.055782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.4.219.131:41149)
0925 22:50:04.056467 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.056479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.057106 (+ 627us) server_negotiation.cc:407] Connection header received
0925 22:50:04.057191 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.057198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.057421 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.057581 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.057866 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059149 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.060895 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.061704 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.061839 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.062019 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.062232 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.062320 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":111,"threads_started":1}
I20260925 22:50:04.063127 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.055771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44767 (local address 127.0.0.1:41672)
0925 22:50:04.056778 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.056815 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.056957 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.057735 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.057738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.057755 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.057922 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.057929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059140 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.059146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.060774 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.060785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.062731 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.062734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.062845 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.062868 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.062946 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.063009 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":173,"spinlock_wait_cycles":8576,"thread_start_us":123,"threads_started":1}
I20260925 22:50:04.063194 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.056232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.4.219.130:44767)
0925 22:50:04.056601 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.056615 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.057100 (+ 485us) server_negotiation.cc:407] Connection header received
0925 22:50:04.057184 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.057190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.057440 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.057582 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.058032 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.059036 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.060915 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.062439 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.062534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.062668 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.062974 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.063060 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260925 22:50:04.065052 12631 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9040b8b624e3f896fa301d504d558 (DEFAULT_TABLE table=test-table [id=774f7d3fd6d1474a9934f3340ed6856e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.066175 12631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9040b8b624e3f896fa301d504d558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.066032 12911 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9040b8b624e3f896fa301d504d558 (DEFAULT_TABLE table=test-table [id=774f7d3fd6d1474a9934f3340ed6856e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.065968 12771 tablet_service.cc:1467] Processing CreateTablet for tablet 56e9040b8b624e3f896fa301d504d558 (DEFAULT_TABLE table=test-table [id=774f7d3fd6d1474a9934f3340ed6856e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.066861 12911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9040b8b624e3f896fa301d504d558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.066928 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e9040b8b624e3f896fa301d504d558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.073205 13005 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Bootstrap starting.
I20260925 22:50:04.074536 13006 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Bootstrap starting.
I20260925 22:50:04.074893 13007 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Bootstrap starting.
I20260925 22:50:04.076575 13005 tablet_bootstrap.cc:654] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.076751 13007 tablet_bootstrap.cc:654] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.076756 13006 tablet_bootstrap.cc:654] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.077458 13007 log.cc:824] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:04.077508 13006 log.cc:824] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:04.077528 13005 log.cc:824] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:04.079329 13007 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: No bootstrap required, opened a new log
I20260925 22:50:04.079404 13005 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: No bootstrap required, opened a new log
I20260925 22:50:04.079517 13006 tablet_bootstrap.cc:492] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: No bootstrap required, opened a new log
I20260925 22:50:04.079592 13007 ts_tablet_manager.cc:1397] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:50:04.079603 13005 ts_tablet_manager.cc:1397] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:50:04.079646 13006 ts_tablet_manager.cc:1397] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:50:04.082445 13007 raft_consensus.cc:348] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.082634 13007 raft_consensus.cc:374] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.082665 13007 raft_consensus.cc:729] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.083101 13007 consensus_queue.cc:260] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.083455 13005 raft_consensus.cc:348] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.083738 13005 raft_consensus.cc:374] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.083773 12837 heartbeater.cc:503] Master 127.4.219.190:45645 was elected leader, sending a full tablet report...
I20260925 22:50:04.083807 13005 raft_consensus.cc:729] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.084157 13007 ts_tablet_manager.cc:1428] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:50:04.084312 13005 consensus_queue.cc:260] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.084299 13006 raft_consensus.cc:348] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.084565 13006 raft_consensus.cc:374] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.084621 13006 raft_consensus.cc:729] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.085219 12977 heartbeater.cc:503] Master 127.4.219.190:45645 was elected leader, sending a full tablet report...
I20260925 22:50:04.085246 13005 ts_tablet_manager.cc:1428] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:50:04.086552 13006 consensus_queue.cc:260] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.087356 13006 ts_tablet_manager.cc:1428] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:50:04.087698 12697 heartbeater.cc:503] Master 127.4.219.190:45645 was elected leader, sending a full tablet report...
W20260925 22:50:04.253018 12978 tablet.cc:2369] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:50:04.289183 12838 tablet.cc:2369] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:50:04.327548 12698 tablet.cc:2369] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:50:04.455717 13011 raft_consensus.cc:482] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:04.455924 13011 raft_consensus.cc:504] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.457103 13011 leader_election.cc:288] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149), b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273)
I20260925 22:50:04.460916 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.457500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:55641 (local address 127.4.219.129:43273)
0925 22:50:04.457627 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.457632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.457802 (+ 170us) server_negotiation.cc:407] Connection header received
0925 22:50:04.457958 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.457960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.458001 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.458083 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.458344 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.458824 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.459956 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.460336 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.460460 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.460513 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.460713 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.460784 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:50:04.460943 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.457490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:37279 (local address 127.4.219.131:41149)
0925 22:50:04.457611 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.457616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.457731 (+ 115us) server_negotiation.cc:407] Connection header received
0925 22:50:04.457893 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.457895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.457941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.458016 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.458288 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.458828 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.459950 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.460364 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.460486 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.460536 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.460742 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.460812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:50:04.460974 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.457322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41149 (local address 127.4.219.130:37279)
0925 22:50:04.457645 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.457663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.457789 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.458039 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.458042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.458056 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.458189 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.458195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.458995 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.458999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.459831 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.459840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.460478 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.460482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.460618 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.460637 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.460727 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.460806 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":48,"thread_start_us":79,"threads_started":1}
I20260925 22:50:04.460981 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.457442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:43273 (local address 127.4.219.130:55641)
0925 22:50:04.457718 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.457729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.457841 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.458093 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.458095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.458110 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.458229 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.458234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.458933 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.458938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.459827 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.459836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.460460 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.460464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.460582 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.460613 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.460697 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.460774 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":44,"thread_start_us":59,"threads_started":1}
I20260925 22:50:04.461846 12931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9040b8b624e3f896fa301d504d558" candidate_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f712049b96fa4cb9b47274f558d2ec70" is_pre_election: true
I20260925 22:50:04.461848 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9040b8b624e3f896fa301d504d558" candidate_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e" is_pre_election: true
I20260925 22:50:04.462167 12651 raft_consensus.cc:2454] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c083e641c0f4cbb8add0bb5f4534d0d in term 0.
I20260925 22:50:04.462167 12931 raft_consensus.cc:2454] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c083e641c0f4cbb8add0bb5f4534d0d in term 0.
I20260925 22:50:04.462657 12725 leader_election.cc:302] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.462915 13011 raft_consensus.cc:2781] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:04.463024 13011 raft_consensus.cc:482] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:04.463093 13011 raft_consensus.cc:3037] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.464140 13011 raft_consensus.cc:504] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.464517 13011 leader_election.cc:288] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [CANDIDATE]: Term 1 election: Requested vote from peers f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149), b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273)
I20260925 22:50:04.464929 12931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9040b8b624e3f896fa301d504d558" candidate_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f712049b96fa4cb9b47274f558d2ec70"
I20260925 22:50:04.464929 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e9040b8b624e3f896fa301d504d558" candidate_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e"
I20260925 22:50:04.465062 12931 raft_consensus.cc:3037] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.465062 12651 raft_consensus.cc:3037] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.466437 12651 raft_consensus.cc:2454] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c083e641c0f4cbb8add0bb5f4534d0d in term 1.
I20260925 22:50:04.466763 12723 leader_election.cc:302] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, b14bd284fb094d899426a409f1a4a42e; no voters:
I20260925 22:50:04.466884 12931 raft_consensus.cc:2454] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c083e641c0f4cbb8add0bb5f4534d0d in term 1.
I20260925 22:50:04.467034 13011 raft_consensus.cc:2781] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:04.467252 13011 raft_consensus.cc:686] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 LEADER]: Becoming Leader. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Running, Role: LEADER
I20260925 22:50:04.467623 13011 consensus_queue.cc:237] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } }
I20260925 22:50:04.470952 12494 catalog_manager.cc:5495] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130). New cstate: current_term: 1 leader_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } health_report { overall_health: UNKNOWN } } }
I20260925 22:50:04.517946 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.514178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.4.219.130:44767)
0925 22:50:04.514317 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.514322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.514337 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:04.514486 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.514488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.514541 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.514647 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:04.514897 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.515398 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.516451 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.516722 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.516831 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.517111 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.517563 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.517690 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.517742 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.517796 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:50:04.523765 12651 raft_consensus.cc:1261] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Refusing update from remote peer 9c083e641c0f4cbb8add0bb5f4534d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.523765 12931 raft_consensus.cc:1261] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Refusing update from remote peer 9c083e641c0f4cbb8add0bb5f4534d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.524459 13016 consensus_queue.cc:1035] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.524703 13011 consensus_queue.cc:1035] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.532403 13020 mvcc.cc:205] Tried to move back new op lower bound from 7333382572122058752 to 7333382571901972480. Current Snapshot: MvccSnapshot[applied={T|T < 7333382572122058752}]
I20260925 22:50:04.539409 12493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:04.539561 12493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:04.541911 12493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Sending DeleteTablet for 3 replicas of tablet 56e9040b8b624e3f896fa301d504d558
I20260925 22:50:04.542594 12911 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e9040b8b624e3f896fa301d504d558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:53812
I20260925 22:50:04.542773 12631 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e9040b8b624e3f896fa301d504d558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:33122
I20260925 22:50:04.543057 12771 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e9040b8b624e3f896fa301d504d558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:41672
I20260925 22:50:04.543390 13029 tablet_replica.cc:330] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: stopping tablet replica
I20260925 22:50:04.543641 13028 tablet_replica.cc:330] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: stopping tablet replica
I20260925 22:50:04.543640 12493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:50:04.543943 13028 raft_consensus.cc:2229] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 22:50:04.544050 12493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:04.544258 13028 raft_consensus.cc:2258] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.545264 13029 raft_consensus.cc:2229] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:04.545557 13029 raft_consensus.cc:2258] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.546437 13028 ts_tablet_manager.cc:1905] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.547250 13030 tablet_replica.cc:330] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: stopping tablet replica
I20260925 22:50:04.547551 13030 raft_consensus.cc:2229] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:04.548062 13030 raft_consensus.cc:2258] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.549409 13028 ts_tablet_manager.cc:1918] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:04.549458 12911 tablet_service.cc:1467] Processing CreateTablet for tablet 4db329b33f1243c586268f80cfdc82e4 (DEFAULT_TABLE table=test-table [id=0d849d60f73642b3a51fa097dd1d41dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.549538 13028 log.cc:1196] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/56e9040b8b624e3f896fa301d504d558
I20260925 22:50:04.549719 12911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db329b33f1243c586268f80cfdc82e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.549850 13028 ts_tablet_manager.cc:1939] T 56e9040b8b624e3f896fa301d504d558 P f712049b96fa4cb9b47274f558d2ec70: Deleting consensus metadata
I20260925 22:50:04.550781 13030 ts_tablet_manager.cc:1905] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.551018 12481 catalog_manager.cc:4841] TS f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149): tablet 56e9040b8b624e3f896fa301d504d558 (table test-table [id=774f7d3fd6d1474a9934f3340ed6856e]) successfully deleted
I20260925 22:50:04.551759 13005 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Bootstrap starting.
I20260925 22:50:04.552196 12771 tablet_service.cc:1467] Processing CreateTablet for tablet 4db329b33f1243c586268f80cfdc82e4 (DEFAULT_TABLE table=test-table [id=0d849d60f73642b3a51fa097dd1d41dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.552433 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db329b33f1243c586268f80cfdc82e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.552731 13005 tablet_bootstrap.cc:654] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.554103 13005 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: No bootstrap required, opened a new log
I20260925 22:50:04.554194 13005 ts_tablet_manager.cc:1397] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:04.554675 13007 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Bootstrap starting.
I20260925 22:50:04.554673 13005 raft_consensus.cc:348] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.554781 13005 raft_consensus.cc:374] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.554813 13005 raft_consensus.cc:729] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.554932 13005 consensus_queue.cc:260] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.555197 13005 ts_tablet_manager.cc:1428] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:04.555146 12631 tablet_service.cc:1467] Processing CreateTablet for tablet 4db329b33f1243c586268f80cfdc82e4 (DEFAULT_TABLE table=test-table [id=0d849d60f73642b3a51fa097dd1d41dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.555390 12631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4db329b33f1243c586268f80cfdc82e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.555665 13007 tablet_bootstrap.cc:654] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.556720 13030 ts_tablet_manager.cc:1918] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:04.556872 13030 log.cc:1196] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/56e9040b8b624e3f896fa301d504d558
I20260925 22:50:04.557288 13030 ts_tablet_manager.cc:1939] T 56e9040b8b624e3f896fa301d504d558 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting consensus metadata
I20260925 22:50:04.557782 13006 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Bootstrap starting.
I20260925 22:50:04.558511 12478 catalog_manager.cc:4841] TS 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767): tablet 56e9040b8b624e3f896fa301d504d558 (table test-table [id=774f7d3fd6d1474a9934f3340ed6856e]) successfully deleted
I20260925 22:50:04.558697 13006 tablet_bootstrap.cc:654] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.559870 13006 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: No bootstrap required, opened a new log
I20260925 22:50:04.559954 13006 ts_tablet_manager.cc:1397] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:04.560389 13006 raft_consensus.cc:348] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.560514 13006 raft_consensus.cc:374] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.560580 13006 raft_consensus.cc:729] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.560662 13029 ts_tablet_manager.cc:1905] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.560711 13006 consensus_queue.cc:260] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.560958 13006 ts_tablet_manager.cc:1428] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:50:04.563777 13029 ts_tablet_manager.cc:1918] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:04.563889 13029 log.cc:1196] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/56e9040b8b624e3f896fa301d504d558
I20260925 22:50:04.564225 13029 ts_tablet_manager.cc:1939] T 56e9040b8b624e3f896fa301d504d558 P b14bd284fb094d899426a409f1a4a42e: Deleting consensus metadata
I20260925 22:50:04.565416 13007 tablet_bootstrap.cc:492] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: No bootstrap required, opened a new log
I20260925 22:50:04.565425 12479 catalog_manager.cc:4841] TS b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273): tablet 56e9040b8b624e3f896fa301d504d558 (table test-table [id=774f7d3fd6d1474a9934f3340ed6856e]) successfully deleted
I20260925 22:50:04.565526 13007 ts_tablet_manager.cc:1397] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260925 22:50:04.565917 13007 raft_consensus.cc:348] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.566030 13007 raft_consensus.cc:374] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.566053 13007 raft_consensus.cc:729] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.566205 13007 consensus_queue.cc:260] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.566478 13007 ts_tablet_manager.cc:1428] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:50:04.583014 13012 raft_consensus.cc:482] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:04.583144 13012 raft_consensus.cc:504] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.584246 13012 leader_election.cc:288] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:04.589780 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.584942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45787 (local address 127.4.219.130:44767)
0925 22:50:04.585044 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.585048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.585570 (+ 522us) server_negotiation.cc:407] Connection header received
0925 22:50:04.585716 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.585719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.585760 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.585802 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.586163 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.586913 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.588450 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.589092 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.589186 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.589237 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.589585 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.589662 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:50:04.590242 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.584593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44767 (local address 127.4.219.131:45787)
0925 22:50:04.585504 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.585515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.585619 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.585918 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.585921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.585945 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.586049 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.586054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.587050 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.587053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.588313 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.588322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.589347 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.589350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.589449 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.589470 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.589995 (+ 525us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.590075 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260925 22:50:04.590349 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.584792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:47259 (local address 127.4.219.129:43273)
0925 22:50:04.584902 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.584906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.585268 (+ 362us) server_negotiation.cc:407] Connection header received
0925 22:50:04.585391 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.585394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.585438 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.585512 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.585879 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.586650 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.589106 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.589695 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.589781 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.589824 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.590159 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.590232 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:50:04.590381 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.584448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:43273 (local address 127.4.219.131:47259)
0925 22:50:04.585169 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.585183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.585294 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.585620 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.585624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.585642 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.585764 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.585769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.586785 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.586789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.588958 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.588968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.589910 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.589914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.590029 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.590053 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.590126 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.590191 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":59,"thread_start_us":116,"threads_started":1}
I20260925 22:50:04.590788 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db329b33f1243c586268f80cfdc82e4" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" is_pre_election: true
I20260925 22:50:04.590826 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db329b33f1243c586268f80cfdc82e4" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e" is_pre_election: true
I20260925 22:50:04.590978 12651 raft_consensus.cc:2454] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:04.591023 12791 raft_consensus.cc:2454] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:04.591372 12863 leader_election.cc:302] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, b14bd284fb094d899426a409f1a4a42e, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.591630 13012 raft_consensus.cc:2781] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:04.591687 13012 raft_consensus.cc:482] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:04.591749 13012 raft_consensus.cc:3037] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.592768 13012 raft_consensus.cc:504] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.593199 13012 leader_election.cc:288] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Requested vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:04.593475 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db329b33f1243c586268f80cfdc82e4" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e"
I20260925 22:50:04.593582 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4db329b33f1243c586268f80cfdc82e4" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
I20260925 22:50:04.593616 12651 raft_consensus.cc:3037] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.593696 12791 raft_consensus.cc:3037] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.594762 12651 raft_consensus.cc:2454] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:04.594790 12791 raft_consensus.cc:2454] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:04.595063 12863 leader_election.cc:302] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14bd284fb094d899426a409f1a4a42e, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.595238 13012 raft_consensus.cc:2781] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:04.595477 13012 raft_consensus.cc:686] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 1 LEADER]: Becoming Leader. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Running, Role: LEADER
I20260925 22:50:04.595726 13012 consensus_queue.cc:237] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.598061 12493 catalog_manager.cc:5495] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 reported cstate change: term changed from 0 to 1, leader changed from <none> to f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } health_report { overall_health: HEALTHY } } }
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376603, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.190@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:04.640007 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.630294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45645 (local address 127.0.0.1:42416)
0925 22:50:04.630657 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.630674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.630788 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.631079 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.631082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.631625 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:04.631751 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.631759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.632642 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.632646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.633584 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.633595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.634152 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.634156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.634254 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.634873 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:04.634889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:04.636609 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:04.638404 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.638415 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.638423 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.638655 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.638927 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:04.638932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:04.638934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:04.639013 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:04.639384 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:04.639394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:04.639505 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.639618 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.639781 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
I20260925 22:50:04.640007 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.630394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.4.219.190:45645)
0925 22:50:04.630716 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.630722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.630737 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:04.630909 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.630912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.630953 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.631032 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.631916 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.632509 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.633718 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.634025 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.634113 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.636742 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:04.636746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:04.636748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:04.636767 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:04.638267 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.638752 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.638758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.638759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.638843 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.639095 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.639100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.639102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.639281 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:04.639383 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.639670 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.639847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":264,"thread_start_us":76,"threads_started":1}
I20260925 22:50:04.644305 12493 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:42416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:50:04.645057 12493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:04.649873 12631 tablet_service.cc:1467] Processing CreateTablet for tablet b8ccf4eb4e3e4d19b06f76c21475916f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c37bcca82a9a4dbf8a8c61ade6d65680]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:50:04.650274 12631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ccf4eb4e3e4d19b06f76c21475916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.653234 12771 tablet_service.cc:1467] Processing CreateTablet for tablet b8ccf4eb4e3e4d19b06f76c21475916f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c37bcca82a9a4dbf8a8c61ade6d65680]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:50:04.653429 12911 tablet_service.cc:1467] Processing CreateTablet for tablet b8ccf4eb4e3e4d19b06f76c21475916f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c37bcca82a9a4dbf8a8c61ade6d65680]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:50:04.653606 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ccf4eb4e3e4d19b06f76c21475916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.653776 12911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ccf4eb4e3e4d19b06f76c21475916f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.654596 13006 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: Bootstrap starting.
I20260925 22:50:04.655666 13006 tablet_bootstrap.cc:654] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.656400 13007 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: Bootstrap starting.
I20260925 22:50:04.656913 13005 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: Bootstrap starting.
I20260925 22:50:04.657373 13007 tablet_bootstrap.cc:654] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.657922 13005 tablet_bootstrap.cc:654] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.658912 13007 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: No bootstrap required, opened a new log
I20260925 22:50:04.658974 13006 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: No bootstrap required, opened a new log
I20260925 22:50:04.658998 13007 ts_tablet_manager.cc:1397] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:04.659057 13006 ts_tablet_manager.cc:1397] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 22:50:04.659097 13005 tablet_bootstrap.cc:492] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: No bootstrap required, opened a new log
I20260925 22:50:04.659163 13005 ts_tablet_manager.cc:1397] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:04.659451 13007 raft_consensus.cc:348] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659564 13007 raft_consensus.cc:374] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.659595 13007 raft_consensus.cc:729] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.659546 13005 raft_consensus.cc:348] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659503 13006 raft_consensus.cc:348] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659638 13006 raft_consensus.cc:374] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.659638 13005 raft_consensus.cc:374] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.659672 13005 raft_consensus.cc:729] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.659672 13006 raft_consensus.cc:729] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.659718 13007 consensus_queue.cc:260] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659793 13006 consensus_queue.cc:260] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659793 13005 consensus_queue.cc:260] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.659971 13007 ts_tablet_manager.cc:1428] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:04.660048 13005 ts_tablet_manager.cc:1428] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:04.660048 13006 ts_tablet_manager.cc:1428] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:50:04.660205 13011 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.660254 13012 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.660339 13011 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.660364 13012 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.660059 13013 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.660642 13013 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.787992 13012 raft_consensus.cc:482] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:04.788141 13012 raft_consensus.cc:504] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.788619 13012 leader_election.cc:288] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:04.789047 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ccf4eb4e3e4d19b06f76c21475916f" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" is_pre_election: true
I20260925 22:50:04.789050 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ccf4eb4e3e4d19b06f76c21475916f" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e" is_pre_election: true
I20260925 22:50:04.789204 12791 raft_consensus.cc:2454] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:04.789213 12651 raft_consensus.cc:2454] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:04.789520 12862 leader_election.cc:302] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.789717 13012 raft_consensus.cc:2781] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:04.789788 13012 raft_consensus.cc:482] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:04.789842 13012 raft_consensus.cc:3037] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.790925 13012 raft_consensus.cc:504] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.791311 13012 leader_election.cc:288] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Requested vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:04.791682 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ccf4eb4e3e4d19b06f76c21475916f" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
I20260925 22:50:04.791720 12651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ccf4eb4e3e4d19b06f76c21475916f" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e"
I20260925 22:50:04.791812 12791 raft_consensus.cc:3037] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.791869 12651 raft_consensus.cc:3037] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.793081 12651 raft_consensus.cc:2454] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:04.793085 12791 raft_consensus.cc:2454] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:04.793383 12862 leader_election.cc:302] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.793545 13012 raft_consensus.cc:2781] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:04.793632 13012 raft_consensus.cc:686] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [term 1 LEADER]: Becoming Leader. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Running, Role: LEADER
I20260925 22:50:04.793780 13012 consensus_queue.cc:237] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.794497 13012 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: TxnStatusTablet state changed. Reason: New leader f712049b96fa4cb9b47274f558d2ec70. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.794589 13012 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:50:04.795019 13049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:50:04.795189 13049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:50:04.795641 12493 catalog_manager.cc:5495] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 reported cstate change: term changed from 0 to 1, leader changed from <none> to f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } health_report { overall_health: HEALTHY } } }
I20260925 22:50:04.816242 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.811673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41149 (local address 127.0.0.1:53818)
0925 22:50:04.811908 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.811924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.812023 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.812372 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.812375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.812383 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:50:04.812502 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.812508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.813622 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.813625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.814314 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.814323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.814906 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.814909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.815019 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.815064 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.815994 (+ 930us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.816006 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.816063 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.816130 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260925 22:50:04.816278 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.811805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.4.219.131:41149)
0925 22:50:04.811944 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.811948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.811964 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:04.812175 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.812178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.812242 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.812351 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:04.812637 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.813516 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.814421 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.814760 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.814890 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.815232 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.815883 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.816012 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.816085 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.816142 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:50:04.819051 12651 raft_consensus.cc:1261] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.819269 12791 raft_consensus.cc:1261] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.819584 13036 consensus_queue.cc:1035] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.819795 13012 consensus_queue.cc:1035] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.821419 13011 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: TxnStatusTablet state changed. Reason: New leader f712049b96fa4cb9b47274f558d2ec70. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.821556 13011 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.822721 13036 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.822858 13036 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:50:04.823342 13013 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: TxnStatusTablet state changed. Reason: New leader f712049b96fa4cb9b47274f558d2ec70. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.823360 13021 mvcc.cc:205] Tried to move back new op lower bound from 7333382573333860352 to 7333382573236637696. Current Snapshot: MvccSnapshot[applied={T|T < 7333382573333860352 or (T in {7333382573333860352})}]
I20260925 22:50:04.823483 13013 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.825450 13013 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.825564 13013 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P b14bd284fb094d899426a409f1a4a42e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.827718 13012 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.827824 13012 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P f712049b96fa4cb9b47274f558d2ec70: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:50:04.828444 13011 tablet_replica.cc:439] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } } }
I20260925 22:50:04.828593 13011 tablet_replica.cc:442] T b8ccf4eb4e3e4d19b06f76c21475916f P 9c083e641c0f4cbb8add0bb5f4534d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:50:04.833555 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.830431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.4.219.131:41149)
0925 22:50:04.830624 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.830629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.830640 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:50:04.830680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.830682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.830726 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.830797 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:04.831052 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.831555 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.832444 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.832728 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.832852 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.833125 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.833198 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.833274 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.833383 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.833430 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260925 22:50:04.843995 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.840738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44591 (local address 127.4.219.131:41149)
0925 22:50:04.840849 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.840853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.841015 (+ 162us) server_negotiation.cc:407] Connection header received
0925 22:50:04.841169 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.841171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.841218 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.841297 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:04.841526 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.842017 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.842954 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.843174 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.843278 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.843544 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.843613 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.843682 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.843817 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.843872 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:04.843999 13057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.840612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41149 (local address 127.4.219.131:44591)
0925 22:50:04.840933 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.840947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.841070 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.841303 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.841305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.841330 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:50:04.841430 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.841440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.842112 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.842118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.842846 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.842855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.843287 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.843290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.843386 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.843424 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.843698 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.843703 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.843770 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.843858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260925 22:50:04.850905 12791 raft_consensus.cc:1261] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.850914 12651 raft_consensus.cc:1261] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:04.851243 13036 consensus_queue.cc:1035] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.851415 13054 consensus_queue.cc:1035] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:04.857899 13019 mvcc.cc:205] Tried to move back new op lower bound from 7333382573465731072 to 7333382572426616832. Current Snapshot: MvccSnapshot[applied={T|T < 7333382573465731072}]
I20260925 22:50:04.893483 12494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:04.893653 12494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:04.895823 12494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Sending DeleteTablet for 3 replicas of tablet 4db329b33f1243c586268f80cfdc82e4
I20260925 22:50:04.896374 12631 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db329b33f1243c586268f80cfdc82e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:33122
I20260925 22:50:04.896422 12771 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db329b33f1243c586268f80cfdc82e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:41672
I20260925 22:50:04.896611 13030 tablet_replica.cc:330] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: stopping tablet replica
I20260925 22:50:04.896602 13029 tablet_replica.cc:330] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: stopping tablet replica
I20260925 22:50:04.896724 13030 raft_consensus.cc:2229] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:04.896730 13029 raft_consensus.cc:2229] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:04.896847 13030 raft_consensus.cc:2258] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.896855 13029 raft_consensus.cc:2258] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.897315 12910 tablet_service.cc:1514] Processing DeleteTablet for tablet 4db329b33f1243c586268f80cfdc82e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:04 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:53812
I20260925 22:50:04.897512 13028 tablet_replica.cc:330] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: stopping tablet replica
I20260925 22:50:04.897600 13028 raft_consensus.cc:2229] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:04.897620 13029 ts_tablet_manager.cc:1905] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.897761 13030 ts_tablet_manager.cc:1905] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.897945 13028 raft_consensus.cc:2258] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:04.898624 13028 ts_tablet_manager.cc:1905] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:04.900635 13029 ts_tablet_manager.cc:1918] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:50:04.900701 13029 log.cc:1196] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/4db329b33f1243c586268f80cfdc82e4
I20260925 22:50:04.900960 13029 ts_tablet_manager.cc:1939] T 4db329b33f1243c586268f80cfdc82e4 P b14bd284fb094d899426a409f1a4a42e: Deleting consensus metadata
I20260925 22:50:04.901032 13030 ts_tablet_manager.cc:1918] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:50:04.901109 13030 log.cc:1196] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/4db329b33f1243c586268f80cfdc82e4
I20260925 22:50:04.901376 13030 ts_tablet_manager.cc:1939] T 4db329b33f1243c586268f80cfdc82e4 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting consensus metadata
I20260925 22:50:04.901444 13028 ts_tablet_manager.cc:1918] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:50:04.901510 13028 log.cc:1196] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/4db329b33f1243c586268f80cfdc82e4
I20260925 22:50:04.901661 12479 catalog_manager.cc:4841] TS b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273): tablet 4db329b33f1243c586268f80cfdc82e4 (table test-table [id=0d849d60f73642b3a51fa097dd1d41dd]) successfully deleted
I20260925 22:50:04.901762 13028 ts_tablet_manager.cc:1939] T 4db329b33f1243c586268f80cfdc82e4 P f712049b96fa4cb9b47274f558d2ec70: Deleting consensus metadata
I20260925 22:50:04.902037 12478 catalog_manager.cc:4841] TS 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767): tablet 4db329b33f1243c586268f80cfdc82e4 (table test-table [id=0d849d60f73642b3a51fa097dd1d41dd]) successfully deleted
I20260925 22:50:04.902463 12481 catalog_manager.cc:4841] TS f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149): tablet 4db329b33f1243c586268f80cfdc82e4 (table test-table [id=0d849d60f73642b3a51fa097dd1d41dd]) successfully deleted
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:50:04.924856 13044 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:04.918630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.4.219.190:45645)
0925 22:50:04.918781 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.918786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.918917 (+ 131us) server_negotiation.cc:407] Connection header received
0925 22:50:04.919066 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.919069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.919115 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.919199 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.919892 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.920439 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.921341 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.921574 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.921681 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.924726 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42430: BlockingRecv error: recv got EOF from 127.0.0.1:42430 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:50:04 dist-test-slave-kb1g krb5kdc[12449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM
I20260925 22:50:04.935482 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.926689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.4.219.190:45645)
0925 22:50:04.926810 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.926816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.926925 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:50:04.927056 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.927058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.927096 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.927165 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:04.927884 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.928370 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.929539 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.929745 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.929828 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.932401 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:04.932404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:04.932406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:04.932428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:04.933896 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.934296 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.934300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.934302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.934386 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:04.934644 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:04.934648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:04.934649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:04.934823 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:04.934953 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.935195 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.935313 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:50:04.938083 12494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:04.938510 12494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:04.943100 12631 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (DEFAULT_TABLE table=test-table [id=5e797cc122bb403e824b237e013ef8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.943445 12631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf94f40a9d84301ab777ff1be4f3ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.944866 12910 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (DEFAULT_TABLE table=test-table [id=5e797cc122bb403e824b237e013ef8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.945218 12910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf94f40a9d84301ab777ff1be4f3ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.946076 13006 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Bootstrap starting.
I20260925 22:50:04.947086 13006 tablet_bootstrap.cc:654] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.947628 13005 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Bootstrap starting.
I20260925 22:50:04.948216 12771 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (DEFAULT_TABLE table=test-table [id=5e797cc122bb403e824b237e013ef8da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:04.948514 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf94f40a9d84301ab777ff1be4f3ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:04.948681 13005 tablet_bootstrap.cc:654] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.950039 13005 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: No bootstrap required, opened a new log
I20260925 22:50:04.950155 13005 ts_tablet_manager.cc:1397] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:04.950594 13005 raft_consensus.cc:348] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.950716 13005 raft_consensus.cc:374] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.950759 13005 raft_consensus.cc:729] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.950888 13007 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Bootstrap starting.
I20260925 22:50:04.950896 13005 consensus_queue.cc:260] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.951198 13005 ts_tablet_manager.cc:1428] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:04.951685 13007 tablet_bootstrap.cc:654] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:04.952850 13007 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: No bootstrap required, opened a new log
I20260925 22:50:04.952935 13007 ts_tablet_manager.cc:1397] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:04.953080 13006 tablet_bootstrap.cc:492] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: No bootstrap required, opened a new log
I20260925 22:50:04.953166 13006 ts_tablet_manager.cc:1397] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 22:50:04.953315 13007 raft_consensus.cc:348] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.953439 13007 raft_consensus.cc:374] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.953475 13007 raft_consensus.cc:729] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.953598 13007 consensus_queue.cc:260] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.953661 13006 raft_consensus.cc:348] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.953783 13006 raft_consensus.cc:374] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:04.953820 13006 raft_consensus.cc:729] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Initialized, Role: FOLLOWER
I20260925 22:50:04.953841 13007 ts_tablet_manager.cc:1428] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:04.953962 13006 consensus_queue.cc:260] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.954797 13006 ts_tablet_manager.cc:1428] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:04.969086 13013 raft_consensus.cc:482] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:04.969230 13013 raft_consensus.cc:504] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.970702 13013 leader_election.cc:288] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767), f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149)
I20260925 22:50:04.975268 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.971074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52847 (local address 127.4.219.131:41149)
0925 22:50:04.971300 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.971305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.971426 (+ 121us) server_negotiation.cc:407] Connection header received
0925 22:50:04.971634 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.971638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.971686 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.971763 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.972568 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.973147 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.974162 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.974689 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.974806 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.974857 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.975067 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.975137 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 22:50:04.975271 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.970933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41149 (local address 127.4.219.129:52847)
0925 22:50:04.971349 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.971360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.971486 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.971873 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.971876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.971894 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.972412 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.972420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.973268 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.973271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.974048 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.974056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.974834 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.974838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.974937 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.974962 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.975036 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.975102 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":37,"thread_start_us":95,"threads_started":1}
I20260925 22:50:04.975958 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.971421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:47351 (local address 127.4.219.130:44767)
0925 22:50:04.971559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.971565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.971581 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:04.971632 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.971635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.971686 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.971755 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:04.972088 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.972795 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.974385 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.974968 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.975094 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.975135 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.975779 (+ 644us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.975848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:04.975983 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.970897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44767 (local address 127.4.219.129:47351)
0925 22:50:04.971254 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:04.971270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:04.971373 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:04.971767 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:04.971770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:04.971785 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:04.971978 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.971986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.972917 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.972921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.974265 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:04.974275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.975213 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.975216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:04.975317 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.975551 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 22:50:04.975766 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:50:04.975845 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260925 22:50:04.976192 12931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf94f40a9d84301ab777ff1be4f3ff3" candidate_uuid: "b14bd284fb094d899426a409f1a4a42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f712049b96fa4cb9b47274f558d2ec70" is_pre_election: true
I20260925 22:50:04.976361 12931 raft_consensus.cc:2454] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14bd284fb094d899426a409f1a4a42e in term 0.
I20260925 22:50:04.976682 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf94f40a9d84301ab777ff1be4f3ff3" candidate_uuid: "b14bd284fb094d899426a409f1a4a42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" is_pre_election: true
I20260925 22:50:04.976744 12585 leader_election.cc:302] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14bd284fb094d899426a409f1a4a42e, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:04.976831 12791 raft_consensus.cc:2454] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b14bd284fb094d899426a409f1a4a42e in term 0.
I20260925 22:50:04.977041 13013 raft_consensus.cc:2781] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:04.977099 13013 raft_consensus.cc:482] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:04.977149 13013 raft_consensus.cc:3037] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.978099 13013 raft_consensus.cc:504] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.978456 13013 leader_election.cc:288] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [CANDIDATE]: Term 1 election: Requested vote from peers 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767), f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149)
I20260925 22:50:04.978734 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf94f40a9d84301ab777ff1be4f3ff3" candidate_uuid: "b14bd284fb094d899426a409f1a4a42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
I20260925 22:50:04.978783 12931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf94f40a9d84301ab777ff1be4f3ff3" candidate_uuid: "b14bd284fb094d899426a409f1a4a42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f712049b96fa4cb9b47274f558d2ec70"
I20260925 22:50:04.978869 12791 raft_consensus.cc:3037] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.978906 12931 raft_consensus.cc:3037] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:04.979804 12791 raft_consensus.cc:2454] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14bd284fb094d899426a409f1a4a42e in term 1.
I20260925 22:50:04.980053 12931 raft_consensus.cc:2454] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b14bd284fb094d899426a409f1a4a42e in term 1.
I20260925 22:50:04.980122 12582 leader_election.cc:302] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c083e641c0f4cbb8add0bb5f4534d0d, b14bd284fb094d899426a409f1a4a42e; no voters:
I20260925 22:50:04.980326 13013 raft_consensus.cc:2781] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:04.980587 13013 raft_consensus.cc:686] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 1 LEADER]: Becoming Leader. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Running, Role: LEADER
I20260925 22:50:04.980845 13013 consensus_queue.cc:237] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:04.983209 12492 catalog_manager.cc:5495] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e reported cstate change: term changed from 0 to 1, leader changed from <none> to b14bd284fb094d899426a409f1a4a42e (127.4.219.129). New cstate: current_term: 1 leader_uuid: "b14bd284fb094d899426a409f1a4a42e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } health_report { overall_health: UNKNOWN } } }
I20260925 22:50:04.999284 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:04.995080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.4.219.129:43273)
0925 22:50:04.995188 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:04.995192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:04.995275 (+ 83us) server_negotiation.cc:407] Connection header received
0925 22:50:04.995424 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:04.995427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:04.995481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:04.995574 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:04.995958 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.996629 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.997820 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:04.998048 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:04.998128 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:04.998446 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:04.998908 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:04.999002 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:04.999085 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 22:50:04.999141 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:50:05.002086 12931 raft_consensus.cc:1261] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Refusing update from remote peer b14bd284fb094d899426a409f1a4a42e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:05.002190 12791 raft_consensus.cc:1261] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Refusing update from remote peer b14bd284fb094d899426a409f1a4a42e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:05.002648 13079 consensus_queue.cc:1035] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:05.003078 13079 consensus_queue.cc:1035] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:05.015311 12492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:05.015516 12492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:05.017690 12492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Sending DeleteTablet for 3 replicas of tablet 0bf94f40a9d84301ab777ff1be4f3ff3
I20260925 22:50:05.018198 12631 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:33122
I20260925 22:50:05.018366 13029 tablet_replica.cc:330] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: stopping tablet replica
I20260925 22:50:05.018478 13029 raft_consensus.cc:2229] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:05.018756 13029 raft_consensus.cc:2258] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.019275 12771 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:41672
I20260925 22:50:05.019479 12910 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf94f40a9d84301ab777ff1be4f3ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:53812
I20260925 22:50:05.019510 13029 ts_tablet_manager.cc:1905] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.019659 13030 tablet_replica.cc:330] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: stopping tablet replica
I20260925 22:50:05.019804 13030 raft_consensus.cc:2229] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:05.019932 13030 raft_consensus.cc:2258] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.020092 13028 tablet_replica.cc:330] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: stopping tablet replica
I20260925 22:50:05.020192 13028 raft_consensus.cc:2229] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:05.020300 13028 raft_consensus.cc:2258] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.020877 13028 ts_tablet_manager.cc:1905] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.021144 13030 ts_tablet_manager.cc:1905] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.022032 13029 ts_tablet_manager.cc:1918] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:05.022099 13029 log.cc:1196] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/0bf94f40a9d84301ab777ff1be4f3ff3
I20260925 22:50:05.022370 13029 ts_tablet_manager.cc:1939] T 0bf94f40a9d84301ab777ff1be4f3ff3 P b14bd284fb094d899426a409f1a4a42e: Deleting consensus metadata
I20260925 22:50:05.023012 12479 catalog_manager.cc:4841] TS b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273): tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (table test-table [id=5e797cc122bb403e824b237e013ef8da]) successfully deleted
I20260925 22:50:05.023166 13028 ts_tablet_manager.cc:1918] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:05.023226 13028 log.cc:1196] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/0bf94f40a9d84301ab777ff1be4f3ff3
I20260925 22:50:05.023474 13028 ts_tablet_manager.cc:1939] T 0bf94f40a9d84301ab777ff1be4f3ff3 P f712049b96fa4cb9b47274f558d2ec70: Deleting consensus metadata
I20260925 22:50:05.023507 13030 ts_tablet_manager.cc:1918] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:05.023574 13030 log.cc:1196] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/0bf94f40a9d84301ab777ff1be4f3ff3
I20260925 22:50:05.023808 13030 ts_tablet_manager.cc:1939] T 0bf94f40a9d84301ab777ff1be4f3ff3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting consensus metadata
I20260925 22:50:05.024032 12481 catalog_manager.cc:4841] TS f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149): tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (table test-table [id=5e797cc122bb403e824b237e013ef8da]) successfully deleted
I20260925 22:50:05.024688 12478 catalog_manager.cc:4841] TS 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767): tablet 0bf94f40a9d84301ab777ff1be4f3ff3 (table test-table [id=5e797cc122bb403e824b237e013ef8da]) successfully deleted
I20260925 22:50:05.024688 12492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:05.025220 12492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:05.030225 12631 tablet_service.cc:1467] Processing CreateTablet for tablet e3d793fba185432a80a68556c06ba0e3 (DEFAULT_TABLE table=test-table [id=fe8187ba7b0240a081f55daae0e3f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:05.030491 12631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d793fba185432a80a68556c06ba0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:05.032637 13006 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: Bootstrap starting.
I20260925 22:50:05.033556 13006 tablet_bootstrap.cc:654] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:05.034297 12910 tablet_service.cc:1467] Processing CreateTablet for tablet e3d793fba185432a80a68556c06ba0e3 (DEFAULT_TABLE table=test-table [id=fe8187ba7b0240a081f55daae0e3f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:05.034555 12910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d793fba185432a80a68556c06ba0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:05.036798 13005 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Bootstrap starting.
I20260925 22:50:05.036783 12771 tablet_service.cc:1467] Processing CreateTablet for tablet e3d793fba185432a80a68556c06ba0e3 (DEFAULT_TABLE table=test-table [id=fe8187ba7b0240a081f55daae0e3f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:05.037011 12771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d793fba185432a80a68556c06ba0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:05.037570 13005 tablet_bootstrap.cc:654] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:05.038736 13005 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: No bootstrap required, opened a new log
I20260925 22:50:05.038803 13005 ts_tablet_manager.cc:1397] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:05.039140 13007 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Bootstrap starting.
I20260925 22:50:05.039160 13005 raft_consensus.cc:348] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.039255 13005 raft_consensus.cc:374] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:05.039284 13005 raft_consensus.cc:729] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Initialized, Role: FOLLOWER
I20260925 22:50:05.039405 13005 consensus_queue.cc:260] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.039637 13005 ts_tablet_manager.cc:1428] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:05.039898 13007 tablet_bootstrap.cc:654] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:05.041270 13006 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: No bootstrap required, opened a new log
I20260925 22:50:05.041378 13006 ts_tablet_manager.cc:1397] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 22:50:05.041787 13006 raft_consensus.cc:348] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.041887 13006 raft_consensus.cc:374] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:05.041918 13006 raft_consensus.cc:729] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14bd284fb094d899426a409f1a4a42e, State: Initialized, Role: FOLLOWER
I20260925 22:50:05.042025 13006 consensus_queue.cc:260] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.042280 13006 ts_tablet_manager.cc:1428] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:05.042375 13007 tablet_bootstrap.cc:492] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: No bootstrap required, opened a new log
I20260925 22:50:05.042454 13007 ts_tablet_manager.cc:1397] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:50:05.042819 13007 raft_consensus.cc:348] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.042929 13007 raft_consensus.cc:374] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:05.042960 13007 raft_consensus.cc:729] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c083e641c0f4cbb8add0bb5f4534d0d, State: Initialized, Role: FOLLOWER
I20260925 22:50:05.043087 13007 consensus_queue.cc:260] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.043349 13007 ts_tablet_manager.cc:1428] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:50:05.147600 13036 raft_consensus.cc:482] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:05.147758 13036 raft_consensus.cc:504] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.148239 13036 leader_election.cc:288] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:05.148778 12650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d793fba185432a80a68556c06ba0e3" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e" is_pre_election: true
I20260925 22:50:05.148937 12650 raft_consensus.cc:2454] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:05.148873 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d793fba185432a80a68556c06ba0e3" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" is_pre_election: true
I20260925 22:50:05.149231 12863 leader_election.cc:302] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14bd284fb094d899426a409f1a4a42e, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:05.149387 13036 raft_consensus.cc:2781] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:05.149444 13036 raft_consensus.cc:482] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:05.149459 12791 raft_consensus.cc:2454] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 0.
I20260925 22:50:05.149500 13036 raft_consensus.cc:3037] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:05.150538 13036 raft_consensus.cc:504] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.150882 13036 leader_election.cc:288] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Requested vote from peers b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273), 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767)
I20260925 22:50:05.151206 12650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d793fba185432a80a68556c06ba0e3" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14bd284fb094d899426a409f1a4a42e"
I20260925 22:50:05.151245 12791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d793fba185432a80a68556c06ba0e3" candidate_uuid: "f712049b96fa4cb9b47274f558d2ec70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d"
I20260925 22:50:05.151332 12650 raft_consensus.cc:3037] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:05.151360 12791 raft_consensus.cc:3037] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:05.152602 12791 raft_consensus.cc:2454] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:05.152643 12650 raft_consensus.cc:2454] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f712049b96fa4cb9b47274f558d2ec70 in term 1.
I20260925 22:50:05.152909 12863 leader_election.cc:302] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14bd284fb094d899426a409f1a4a42e, f712049b96fa4cb9b47274f558d2ec70; no voters:
I20260925 22:50:05.153074 13036 raft_consensus.cc:2781] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:05.153179 13036 raft_consensus.cc:686] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 LEADER]: Becoming Leader. State: Replica: f712049b96fa4cb9b47274f558d2ec70, State: Running, Role: LEADER
I20260925 22:50:05.153313 13036 consensus_queue.cc:237] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } }
I20260925 22:50:05.155041 12494 catalog_manager.cc:5495] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 reported cstate change: term changed from 0 to 1, leader changed from <none> to f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "f712049b96fa4cb9b47274f558d2ec70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f712049b96fa4cb9b47274f558d2ec70" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41149 } health_report { overall_health: HEALTHY } } }
I20260925 22:50:05.170888 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:05.167448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.4.219.131:41149)
0925 22:50:05.167552 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:05.167556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:05.167597 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:50:05.167705 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:05.167708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:05.167756 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:05.167841 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:05.168109 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.168917 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.169800 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:05.170110 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:05.170216 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:05.170436 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:05.170512 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:05.170599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:05.170689 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:50:05.170740 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:50:05.176774 12650 raft_consensus.cc:1261] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:05.176791 12790 raft_consensus.cc:1261] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Refusing update from remote peer f712049b96fa4cb9b47274f558d2ec70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:05.177142 13036 consensus_queue.cc:1035] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c083e641c0f4cbb8add0bb5f4534d0d" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:05.177275 13012 consensus_queue.cc:1035] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14bd284fb094d899426a409f1a4a42e" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:05.208393 12492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:05.208559 12492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:05.210489 12492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc5e8c9ebf264b09b5cc2c3b158d5a75: Sending DeleteTablet for 3 replicas of tablet e3d793fba185432a80a68556c06ba0e3
I20260925 22:50:05.211001 12631 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d793fba185432a80a68556c06ba0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:33122
I20260925 22:50:05.211139 12910 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d793fba185432a80a68556c06ba0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:53812
I20260925 22:50:05.211202 13029 tablet_replica.cc:330] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: stopping tablet replica
I20260925 22:50:05.211172 12771 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d793fba185432a80a68556c06ba0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:05 UTC) from {username='oryx', principal='oryx/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:41672
I20260925 22:50:05.211336 13030 tablet_replica.cc:330] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: stopping tablet replica
I20260925 22:50:05.211349 13028 tablet_replica.cc:330] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: stopping tablet replica
I20260925 22:50:05.211324 13029 raft_consensus.cc:2229] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:05.211448 13030 raft_consensus.cc:2229] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:05.211453 13028 raft_consensus.cc:2229] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:05.211468 13029 raft_consensus.cc:2258] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.211565 13030 raft_consensus.cc:2258] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.211676 13028 raft_consensus.cc:2258] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:05.212265 13029 ts_tablet_manager.cc:1905] T e3d793fba185432a80a68556c06ba0e3 P b14bd284fb094d899426a409f1a4a42e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.212276 13030 ts_tablet_manager.cc:1905] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.212369 13028 ts_tablet_manager.cc:1905] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:05.212380 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12566
I20260925 22:50:05.215179 13030 ts_tablet_manager.cc:1918] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:50:05.215250 13030 log.cc:1196] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/e3d793fba185432a80a68556c06ba0e3
I20260925 22:50:05.215504 13030 ts_tablet_manager.cc:1939] T e3d793fba185432a80a68556c06ba0e3 P 9c083e641c0f4cbb8add0bb5f4534d0d: Deleting consensus metadata
I20260925 22:50:05.216212 13028 ts_tablet_manager.cc:1918] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:50:05.216177 12478 catalog_manager.cc:4841] TS 9c083e641c0f4cbb8add0bb5f4534d0d (127.4.219.130:44767): tablet e3d793fba185432a80a68556c06ba0e3 (table test-table [id=fe8187ba7b0240a081f55daae0e3f02f]) successfully deleted
I20260925 22:50:05.216279 13028 log.cc:1196] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/e3d793fba185432a80a68556c06ba0e3
I20260925 22:50:05.216557 13028 ts_tablet_manager.cc:1939] T e3d793fba185432a80a68556c06ba0e3 P f712049b96fa4cb9b47274f558d2ec70: Deleting consensus metadata
I20260925 22:50:05.217151 12481 catalog_manager.cc:4841] TS f712049b96fa4cb9b47274f558d2ec70 (127.4.219.131:41149): tablet e3d793fba185432a80a68556c06ba0e3 (table test-table [id=fe8187ba7b0240a081f55daae0e3f02f]) successfully deleted
W20260925 22:50:05.220568 12479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:43273 (error 108)
I20260925 22:50:05.220753 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12706
I20260925 22:50:05.222955 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:05.222158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:43273 (local address 127.0.0.1:33134)
0925 22:50:05.222755 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:05.222873 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:43273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":83,"threads_started":1}
W20260925 22:50:05.223133 12479 catalog_manager.cc:4584] TS b14bd284fb094d899426a409f1a4a42e (127.4.219.129:43273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3d793fba185432a80a68556c06ba0e3: Network error: Client connection negotiation failed: client connection to 127.4.219.129:43273: connect: Connection refused (error 111)
I20260925 22:50:05.227406 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12846
I20260925 22:50:05.234071 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 12462
2026-09-25T22:50:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3299 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:05 dist-test-slave-kb1g krb5kdc[13095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:05 dist-test-slave-kb1g krb5kdc[13095](info): set up 2 sockets
Sep 25 22:50:05 dist-test-slave-kb1g krb5kdc[13095](info): commencing operation
krb5kdc: starting...
W20260925 22:50:06.273274 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:50:06 dist-test-slave-kb1g krb5kdc[13095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:50:06.391373 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36509
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:38183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:36509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:06.503494 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:06.503756 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:06.503798 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:06.508606 13108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:06.508669 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:06.508689 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:06.508706 13108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:06.508721 13108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:06.513964 13108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:38183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:36509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:06.514964 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:06.515942 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:06.524575 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:06.524634 13108 server_base.cc:1034] running on GCE node
W20260925 22:50:06.524585 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:06.524580 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:06.525328 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:06.526487 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:06.527673 13108 hybrid_clock.cc:648] HybridClock initialized: now 1790376606527657 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:50:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1303 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:06 dist-test-slave-kb1g krb5kdc[13121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:06 dist-test-slave-kb1g krb5kdc[13121](info): set up 2 sockets
Sep 25 22:50:06 dist-test-slave-kb1g krb5kdc[13121](info): commencing operation
krb5kdc: starting...
W20260925 22:50:07.601233 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:50:07 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:50:07.719695 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45213
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:07.826927 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:07.827196 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:07.827236 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:07.831143 13134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:07.831234 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:07.831275 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:07.831323 13134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:07.831355 13134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:07.836496 13134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:07.837559 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:07.838586 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:07.846820 13139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:07.846999 13134 server_base.cc:1034] running on GCE node
W20260925 22:50:07.849128 13142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:07.849375 13140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:07.849941 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:07.851367 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:07.852690 13134 hybrid_clock.cc:648] HybridClock initialized: now 1790376607852656 us; error 45 us; skew 500 ppm
Sep 25 22:50:07 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376607, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:07.855510 13134 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:50:07.856408 13134 webserver.cc:460] Webserver started at http://127.4.219.190:33597/ using document root <none> and password file <none>
I20260925 22:50:07.856993 13134 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:07.857056 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:07.857270 13134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:07.859504 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "23608df1e7204bb29975af822d6c1459"
format_stamp: "Formatted at 2026-09-25 22:50:07 on dist-test-slave-kb1g"
server_key: "49f2ccab6ed3f0bcb32694cf17c3c2b5"
server_key_iv: "54c69111567286ff80022a20b1176b73"
server_key_version: "encryptionkey@0"
I20260925 22:50:07.860118 13134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "23608df1e7204bb29975af822d6c1459"
format_stamp: "Formatted at 2026-09-25 22:50:07 on dist-test-slave-kb1g"
server_key: "49f2ccab6ed3f0bcb32694cf17c3c2b5"
server_key_iv: "54c69111567286ff80022a20b1176b73"
server_key_version: "encryptionkey@0"
I20260925 22:50:07.863921 13134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:50:07.866361 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:07.867596 13134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:07.867725 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "23608df1e7204bb29975af822d6c1459"
format_stamp: "Formatted at 2026-09-25 22:50:07 on dist-test-slave-kb1g"
server_key: "49f2ccab6ed3f0bcb32694cf17c3c2b5"
server_key_iv: "54c69111567286ff80022a20b1176b73"
server_key_version: "encryptionkey@0"
I20260925 22:50:07.867859 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:07.877352 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:07.880280 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:07.880486 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:07.887442 13134 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:45213
I20260925 22:50:07.887470 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:45213 every 8 connection(s)
I20260925 22:50:07.888337 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:07.891132 13202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:07.895496 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13134
I20260925 22:50:07.895628 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:50:07.895891 4974 external_mini_cluster.cc:1428] Setting key 63d8e68144f9da96990cbee53de9e89f
I20260925 22:50:07.896833 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Bootstrap starting.
I20260925 22:50:07.899433 13202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:07.900247 13202 log.cc:824] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:07.902249 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: No bootstrap required, opened a new log
Sep 25 22:50:07 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:07.906026 13202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:07.906288 13202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:07.906378 13202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23608df1e7204bb29975af822d6c1459, State: Initialized, Role: FOLLOWER
I20260925 22:50:07.906853 13202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:07.907007 13202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:07.907073 13202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:07.907167 13202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:07.908056 13202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:07.908393 13202 leader_election.cc:302] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23608df1e7204bb29975af822d6c1459; no voters:
I20260925 22:50:07.908730 13202 leader_election.cc:288] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:07.908825 13207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:07.909070 13207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 LEADER]: Becoming Leader. State: Replica: 23608df1e7204bb29975af822d6c1459, State: Running, Role: LEADER
I20260925 22:50:07.909461 13207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:07.909952 13202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:07.911119 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:07.897302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.4.219.190:45213)
0925 22:50:07.897694 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:07.897713 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:07.897756 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:50:07.898602 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:07.898625 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:07.899036 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:07.899392 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:07.900190 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:07.901561 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:07.902401 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:07.902872 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:07.903043 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:07.905529 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:07.905534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:07.905546 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:07.905571 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:07.907458 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:07.907929 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:07.907934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:07.907938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:07.908051 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:07.908339 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:07.908343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:07.908345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:07.908903 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:07.909697 (+ 794us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:07.909844 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 22:50:07.910142 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":101,"threads_started":1}
I20260925 22:50:07.911330 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23608df1e7204bb29975af822d6c1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } } }
I20260925 22:50:07.911453 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:07.911677 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23608df1e7204bb29975af822d6c1459. Latest consensus state: current_term: 1 leader_uuid: "23608df1e7204bb29975af822d6c1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } } }
I20260925 22:50:07.911775 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:07.911914 13217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:07.914991 13217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:07.920125 13217 catalog_manager.cc:1347] Generated new cluster ID: f8b7f18abb984cd8b405406ddb7ec3b5
I20260925 22:50:07.920260 13217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:07.932610 13217 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:07.933530 13217 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:07.941087 13217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Generated new TSK 0
I20260925 22:50:07.941740 13217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:50:07.973614 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:08.083591 13228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:08.083948 13228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:08.084075 13228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:08.088119 13228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:08.088173 13228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:08.088235 13228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:08.093688 13228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:08.094785 13228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:08.095801 13228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:08.105168 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.105160 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.105160 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.105434 13228 server_base.cc:1034] running on GCE node
I20260925 22:50:08.105868 13228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:08.106462 13228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:08.107621 13228 hybrid_clock.cc:648] HybridClock initialized: now 1790376608107607 us; error 25 us; skew 500 ppm
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:08.110405 13228 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:50:08.111436 13228 webserver.cc:460] Webserver started at http://127.4.219.129:41805/ using document root <none> and password file <none>
I20260925 22:50:08.112023 13228 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:08.112103 13228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:08.112356 13228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:08.114455 13228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "45a2a288fd2444e59a82c0f72201c53d"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "3fbf77f329634e5b9150e35a981754b9"
server_key_iv: "ea85bfb2eb23c05d7c016d6750181066"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.115012 13228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "45a2a288fd2444e59a82c0f72201c53d"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "3fbf77f329634e5b9150e35a981754b9"
server_key_iv: "ea85bfb2eb23c05d7c016d6750181066"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.118814 13228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:50:08.121343 13243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.122519 13228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:08.122620 13228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "45a2a288fd2444e59a82c0f72201c53d"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "3fbf77f329634e5b9150e35a981754b9"
server_key_iv: "ea85bfb2eb23c05d7c016d6750181066"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.122702 13228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:08.148501 13228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:08.151343 13228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:08.151494 13228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:08.152015 13228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:08.152819 13228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:08.152873 13228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.152915 13228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:08.152930 13228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.161779 13228 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:40805
I20260925 22:50:08.161832 13356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:40805 every 8 connection(s)
I20260925 22:50:08.162684 13228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:08.169687 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13228
I20260925 22:50:08.169850 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:50:08.170176 4974 external_mini_cluster.cc:1428] Setting key 15955dd903496471bb7ac970b23d7e93
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:08.177289 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.164457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:60011 (local address 127.4.219.190:45213)
0925 22:50:08.164635 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:08.164639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:08.165448 (+ 809us) server_negotiation.cc:407] Connection header received
0925 22:50:08.166727 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:08.166730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:08.166784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:08.166865 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:08.167964 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.168500 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.169778 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.170068 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.170156 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.173055 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:08.173062 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:08.173068 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:08.173101 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:08.175002 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.175628 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.175635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.175640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.175735 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.176048 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.176054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.176058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.176310 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:08.176417 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:08.176964 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 22:50:08.177095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:50:08.178051 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.164879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:60011)
0925 22:50:08.165291 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:08.165333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:08.166524 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:08.167029 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:08.167036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:08.167369 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:08.167818 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.167830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.168656 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.168659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.169636 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.169649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.170221 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.170225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.170423 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.171112 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:08.171128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:08.172887 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:08.175118 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.175127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.175148 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.175517 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.175825 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.175828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.175830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.175959 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.176417 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:08.176423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:08.176710 (+ 287us) client_negotiation.cc:769] Sending connection context
0925 22:50:08.176886 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 22:50:08.177138 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260925 22:50:08.179325 13357 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45213
I20260925 22:50:08.179570 13357 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:08.180052 13357 heartbeater.cc:511] Master 127.4.219.190:45213 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:50:08.181610 13166 ts_manager.cc:194] Registered new tserver with Master: 45a2a288fd2444e59a82c0f72201c53d (127.4.219.129:40805)
I20260925 22:50:08.182722 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:60011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:50:08.199081 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:08.309322 13362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:08.309573 13362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:08.309664 13362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:08.313818 13362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:08.313880 13362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:08.313938 13362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:08.319245 13362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:08.320281 13362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:08.321357 13362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:08.330924 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.330971 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.331226 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.331189 13362 server_base.cc:1034] running on GCE node
I20260925 22:50:08.331637 13362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:08.332257 13362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:08.333437 13362 hybrid_clock.cc:648] HybridClock initialized: now 1790376608333402 us; error 45 us; skew 500 ppm
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:08.336216 13362 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:50:08.337149 13362 webserver.cc:460] Webserver started at http://127.4.219.130:38441/ using document root <none> and password file <none>
I20260925 22:50:08.337683 13362 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:08.337740 13362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:08.337913 13362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:08.340317 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "70263ee874eb4fbb89e3cbbd5b360933"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "e7bf95ee471b3ecd00735c320d1ed31c"
server_key_iv: "761d42529089d974e7589e32334ed565"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.340883 13362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "70263ee874eb4fbb89e3cbbd5b360933"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "e7bf95ee471b3ecd00735c320d1ed31c"
server_key_iv: "761d42529089d974e7589e32334ed565"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.344434 13362 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:50:08.346752 13377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.347980 13362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:08.348100 13362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "70263ee874eb4fbb89e3cbbd5b360933"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "e7bf95ee471b3ecd00735c320d1ed31c"
server_key_iv: "761d42529089d974e7589e32334ed565"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.348201 13362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:08.372037 13362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:08.374962 13362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:08.375170 13362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:08.375775 13362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:08.376600 13362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:08.376643 13362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.376708 13362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:08.376746 13362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.385716 13362 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:36099
I20260925 22:50:08.385735 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:36099 every 8 connection(s)
I20260925 22:50:08.386607 13362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:50:08.395370 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13362
I20260925 22:50:08.395498 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:08.395803 4974 external_mini_cluster.cc:1428] Setting key cd95bfc46d3114e72a5976182734f936
I20260925 22:50:08.400060 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.388311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:40545 (local address 127.4.219.190:45213)
0925 22:50:08.388450 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:08.388454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:08.389334 (+ 880us) server_negotiation.cc:407] Connection header received
0925 22:50:08.390592 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:08.390596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:08.390652 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:08.390736 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:08.391862 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.392398 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.393565 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.393791 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.393898 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.396319 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:08.396323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:08.396326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:08.396354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:08.397827 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.398410 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.398412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.398413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.398483 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.398877 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.398882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.398885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.399140 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:08.399242 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:08.399689 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 22:50:08.399834 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:50:08.400753 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.388733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:40545)
0925 22:50:08.389192 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:08.389235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:08.390375 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:08.390909 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:08.390916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:08.391257 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:08.391722 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.391735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.392517 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.392521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.393441 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.393450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.393930 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.393934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.394101 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.394637 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:08.394652 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:08.396165 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:08.397937 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.397946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.397963 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.398316 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.398564 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.398570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.398575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.398761 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.399241 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:08.399246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:08.399466 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 22:50:08.399615 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 22:50:08.399843 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
I20260925 22:50:08.401818 13491 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45213
I20260925 22:50:08.402040 13491 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:08.402493 13491 heartbeater.cc:511] Master 127.4.219.190:45213 requested a full tablet report, sending...
I20260925 22:50:08.403455 13166 ts_manager.cc:194] Registered new tserver with Master: 70263ee874eb4fbb89e3cbbd5b360933 (127.4.219.130:36099)
I20260925 22:50:08.404008 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:40545
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:50:08.423980 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:08.535029 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:08.535267 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:08.535308 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:08.539206 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:08.539275 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:08.539333 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:08.544416 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:08.545399 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:08.546331 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:08.555667 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.555682 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.555680 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.556057 13496 server_base.cc:1034] running on GCE node
I20260925 22:50:08.556421 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:08.557052 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:08.558192 13496 hybrid_clock.cc:648] HybridClock initialized: now 1790376608558178 us; error 29 us; skew 500 ppm
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:08.561005 13496 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:50:08.561815 13496 webserver.cc:460] Webserver started at http://127.4.219.131:45809/ using document root <none> and password file <none>
I20260925 22:50:08.562350 13496 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:08.562434 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:08.562657 13496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:08.564714 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "dcbc59460dc643c49d22eb4bdedb7baa"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "ad96648b856beffdc8f14815bf48932f"
server_key_iv: "3656d7cde1c0213347984b15584ecf77"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.565260 13496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "dcbc59460dc643c49d22eb4bdedb7baa"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "ad96648b856beffdc8f14815bf48932f"
server_key_iv: "3656d7cde1c0213347984b15584ecf77"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.568840 13496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:50:08.571198 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.572468 13496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:08.572621 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "dcbc59460dc643c49d22eb4bdedb7baa"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "ad96648b856beffdc8f14815bf48932f"
server_key_iv: "3656d7cde1c0213347984b15584ecf77"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.572710 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:08.594103 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:08.597064 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:08.597218 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:08.597746 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:08.598495 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:08.598541 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.598582 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:08.598603 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.606906 13496 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:37035
I20260925 22:50:08.606931 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:37035 every 8 connection(s)
I20260925 22:50:08.607784 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:50:08.610437 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13496
I20260925 22:50:08.610586 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:50:08.610888 4974 external_mini_cluster.cc:1428] Setting key 87bc4ea1af41c5d7e2db623f9562b905
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:08.621301 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.609654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:58581 (local address 127.4.219.190:45213)
0925 22:50:08.609799 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:08.609803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:08.610702 (+ 899us) server_negotiation.cc:407] Connection header received
0925 22:50:08.612328 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:08.612332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:08.612380 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:08.612464 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:08.613743 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.614265 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.615380 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.615586 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.615658 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.618074 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:08.618077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:08.618079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:08.618102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:08.619372 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.619879 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.619881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.619883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.619966 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:08.620334 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:08.620336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:08.620337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:08.620495 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:08.620631 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:08.621086 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 22:50:08.621185 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:50:08.622146 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:08.610035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:58581)
0925 22:50:08.610517 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:08.610604 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:08.612106 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:08.612668 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:08.612675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:08.613136 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:08.613618 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.613631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.614367 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.614370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.615276 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:08.615285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:08.615704 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:08.615707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:08.615884 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:08.616483 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:08.616499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:08.617937 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:08.619493 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.619497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.619509 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.619791 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.620116 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:08.620119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:08.620120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:08.620258 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:08.620618 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:08.620623 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:08.620868 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 22:50:08.621027 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 22:50:08.621290 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":126,"threads_started":1}
I20260925 22:50:08.623461 13625 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45213
I20260925 22:50:08.623703 13625 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:08.624293 13625 heartbeater.cc:511] Master 127.4.219.190:45213 requested a full tablet report, sending...
I20260925 22:50:08.625337 13166 ts_manager.cc:194] Registered new tserver with Master: dcbc59460dc643c49d22eb4bdedb7baa (127.4.219.131:37035)
I20260925 22:50:08.625998 13166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:58581
I20260925 22:50:08.636248 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
Entry for principal oryx/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab.
WARNING: no policy specified for oryx/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
Entry for principal oryx/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab.
WARNING: no policy specified for oryx/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
Entry for principal oryx/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab.
I20260925 22:50:08.707239 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13228
I20260925 22:50:08.713147 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13362
I20260925 22:50:08.718469 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13496
I20260925 22:50:08.723702 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13134
I20260925 22:50:08.729779 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45213
--webserver_interface=127.4.219.190
--webserver_port=33597
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:08.845093 13634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:08.845359 13634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:08.845402 13634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:08.849611 13634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:08.849679 13634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:08.849699 13634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:08.849716 13634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:08.849731 13634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:08.855655 13634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=33597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:08.856873 13634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:08.857946 13634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:08.866442 13640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.866559 13634 server_base.cc:1034] running on GCE node
W20260925 22:50:08.866431 13639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:08.866432 13642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:08.867281 13634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:08.868307 13634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:08.869464 13634 hybrid_clock.cc:648] HybridClock initialized: now 1790376608869427 us; error 50 us; skew 500 ppm
Sep 25 22:50:08 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376608, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:08.872159 13634 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:50:08.873060 13634 webserver.cc:460] Webserver started at http://127.4.219.190:33597/ using document root <none> and password file <none>
I20260925 22:50:08.873601 13634 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:08.873703 13634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:08.877403 13634 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:08.879357 13649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:08.880702 13634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:08.880844 13634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "23608df1e7204bb29975af822d6c1459"
format_stamp: "Formatted at 2026-09-25 22:50:07 on dist-test-slave-kb1g"
server_key: "49f2ccab6ed3f0bcb32694cf17c3c2b5"
server_key_iv: "54c69111567286ff80022a20b1176b73"
server_key_version: "encryptionkey@0"
I20260925 22:50:08.881420 13634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:08.899874 13634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:08.903573 13634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:08.903787 13634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:08.911008 13634 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:45213
I20260925 22:50:08.911021 13701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:45213 every 8 connection(s)
I20260925 22:50:08.912052 13634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:08.916242 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13634
I20260925 22:50:08.916371 13702 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:50:08.916785 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:40805
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:45213
--webserver_port=41805
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:50:08.918706 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Bootstrap starting.
I20260925 22:50:08.935292 13702 log.cc:824] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:08.941778 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:50:08.942144 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Bootstrap complete.
I20260925 22:50:08.946312 13702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:08.946691 13702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23608df1e7204bb29975af822d6c1459, State: Initialized, Role: FOLLOWER
I20260925 22:50:08.947165 13702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:08.947274 13702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:08.947355 13702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:08.947445 13702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:50:08.949375 13702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:08.949697 13702 leader_election.cc:302] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23608df1e7204bb29975af822d6c1459; no voters:
I20260925 22:50:08.950014 13702 leader_election.cc:288] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:50:08.950119 13706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:50:08.950315 13706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [term 2 LEADER]: Becoming Leader. State: Replica: 23608df1e7204bb29975af822d6c1459, State: Running, Role: LEADER
I20260925 22:50:08.950608 13706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } }
I20260925 22:50:08.951195 13702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:08.951604 13706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23608df1e7204bb29975af822d6c1459. Latest consensus state: current_term: 2 leader_uuid: "23608df1e7204bb29975af822d6c1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } } }
I20260925 22:50:08.951707 13706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:08.952164 13707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23608df1e7204bb29975af822d6c1459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23608df1e7204bb29975af822d6c1459" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45213 } } }
I20260925 22:50:08.952468 13707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:08.953507 13715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:08.956270 13715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:08.956866 13715 catalog_manager.cc:1259] Loaded cluster ID: f8b7f18abb984cd8b405406ddb7ec3b5
I20260925 22:50:08.956943 13715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:08.958559 13715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:08.959113 13715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23608df1e7204bb29975af822d6c1459: Loaded TSK: 0
I20260925 22:50:08.960378 13715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:50:09.050128 13704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:09.050407 13704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:09.050485 13704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:09.055004 13704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:09.055090 13704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:09.055176 13704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:09.060438 13704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.129.keytab
--principal=oryx/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:40805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=41805
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:09.061496 13704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:09.062505 13704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:09.071913 13728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.071925 13729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.072093 13731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.072229 13704 server_base.cc:1034] running on GCE node
I20260925 22:50:09.072767 13704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:09.073450 13704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:09.074622 13704 hybrid_clock.cc:648] HybridClock initialized: now 1790376609074611 us; error 29 us; skew 500 ppm
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:09.077400 13704 init.cc:377] Logged in from keytab as oryx/127.4.219.129@KRBTEST.COM (short username oryx)
I20260925 22:50:09.078315 13704 webserver.cc:460] Webserver started at http://127.4.219.129:41805/ using document root <none> and password file <none>
I20260925 22:50:09.078852 13704 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:09.078922 13704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:09.082738 13704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:50:09.084884 13738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.086186 13704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:50:09.086308 13704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "45a2a288fd2444e59a82c0f72201c53d"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "3fbf77f329634e5b9150e35a981754b9"
server_key_iv: "ea85bfb2eb23c05d7c016d6750181066"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.086839 13704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:09.119244 13704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:09.122359 13704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:09.122562 13704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:09.123198 13704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:09.124035 13704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:09.124081 13704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.124148 13704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:09.124178 13704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.133176 13704 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:40805
I20260925 22:50:09.133200 13851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:40805 every 8 connection(s)
I20260925 22:50:09.134167 13704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:09.144029 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13704
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.144505 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:36099
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:45213
--webserver_port=38441
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.146908 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.136287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:53827)
0925 22:50:09.136762 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.136813 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.137985 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.138972 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.138979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.139323 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.139798 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.139811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.141072 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.141075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.141908 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.141916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.142694 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.142697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.142847 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.143439 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.143455 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.146178 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":139,"threads_started":1}
W20260925 22:50:09.147410 13852 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:45213 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.219.190:45213: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
W20260925 22:50:09.147933 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.136231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:53827 (local address 127.4.219.190:45213)
0925 22:50:09.136690 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.136705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.136923 (+ 218us) server_negotiation.cc:407] Connection header received
0925 22:50:09.138324 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.138336 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.138617 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.138893 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.139942 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.140959 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.142031 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.142526 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.142646 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.146494 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:53827: BlockingRecv error: recv got EOF from 127.4.219.129:53827 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":116,"threads_started":1}
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.153996 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.147876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:44163)
0925 22:50:09.148212 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.148228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.148305 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.148631 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.148634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.148854 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.148944 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.148949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.150062 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.150065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.150596 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.150604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.151232 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.151234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.151301 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.151904 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.151920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.153909 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 22:50:09.154270 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.147935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:44163 (local address 127.4.219.190:45213)
0925 22:50:09.148244 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.148250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.148269 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:50:09.148422 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.148428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.148489 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.148625 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.149061 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.149948 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.150710 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.151124 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.151250 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.154098 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:44163: BlockingRecv error: recv got EOF from 127.4.219.129:44163 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.159879 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.154486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:43977)
0925 22:50:09.154609 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.154627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.154707 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.155008 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.155014 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.155220 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.155330 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.155340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.156142 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.156149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.157071 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.157082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.157482 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.157485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.157567 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.157988 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.158001 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.159764 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:50:09.160012 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.154703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43977 (local address 127.4.219.190:45213)
0925 22:50:09.154811 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.154815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.154836 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:50:09.154864 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.154866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.154898 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.154960 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.155475 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.156020 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.157178 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.157383 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.157460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.159880 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:43977: BlockingRecv error: recv got EOF from 127.4.219.129:43977 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 22:50:09.160202 13852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:50:09.258206 13856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:09.258460 13856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:09.258502 13856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:09.262507 13856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:09.262570 13856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:09.262658 13856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:09.267923 13856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.130.keytab
--principal=oryx/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:36099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=38441
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:09.269023 13856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:09.270035 13856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:09.279580 13863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.279587 13856 server_base.cc:1034] running on GCE node
W20260925 22:50:09.279608 13862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.279608 13865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.280171 13856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:09.280889 13856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:09.282070 13856 hybrid_clock.cc:648] HybridClock initialized: now 1790376609282041 us; error 42 us; skew 500 ppm
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:09.284997 13856 init.cc:377] Logged in from keytab as oryx/127.4.219.130@KRBTEST.COM (short username oryx)
I20260925 22:50:09.285830 13856 webserver.cc:460] Webserver started at http://127.4.219.130:38441/ using document root <none> and password file <none>
I20260925 22:50:09.286398 13856 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:09.286491 13856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:09.290412 13856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:09.292759 13872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.294086 13856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:09.294246 13856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "70263ee874eb4fbb89e3cbbd5b360933"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "e7bf95ee471b3ecd00735c320d1ed31c"
server_key_iv: "761d42529089d974e7589e32334ed565"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.294847 13856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:09.307431 13856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:09.310622 13856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:09.310832 13856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:09.311481 13856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:09.312294 13856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:09.312367 13856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.312438 13856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:09.312480 13856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.321197 13856 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:36099
I20260925 22:50:09.321223 13985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:36099 every 8 connection(s)
I20260925 22:50:09.322326 13856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.331921 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13856
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.332455 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:37035
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:45213
--webserver_port=45809
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:09.333824 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.324196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:42297 (local address 127.4.219.190:45213)
0925 22:50:09.324343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.324347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.325296 (+ 949us) server_negotiation.cc:407] Connection header received
0925 22:50:09.326524 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.326527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.326580 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.326658 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.327852 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.328402 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.329574 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.329784 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.329872 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.333720 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:42297: BlockingRecv error: recv got EOF from 127.4.219.130:42297 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:50:09.334573 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.324686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:42297)
0925 22:50:09.325137 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.325180 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.326317 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.326836 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.326842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.327220 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.327706 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.327719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.328519 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.328522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.329469 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.329478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.329930 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.329933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.330071 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.330721 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.330738 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.333597 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
W20260925 22:50:09.335163 13986 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:45213 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.219.190:45213: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.341441 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.335659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:39957)
0925 22:50:09.335780 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.335796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.335897 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.336138 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.336140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.336271 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.336361 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.336366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.337504 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.337507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.338084 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.338092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.338762 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.338765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.338862 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.339430 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.339442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.341342 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:50:09.341719 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.335764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:39957 (local address 127.4.219.190:45213)
0925 22:50:09.335898 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.335904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.335922 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:50:09.335968 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.335974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.336027 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.336120 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.336476 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.337352 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.338214 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.338640 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.338768 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.341549 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:39957: BlockingRecv error: recv got EOF from 127.4.219.130:39957 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.347216 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.342045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:36917)
0925 22:50:09.342146 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.342162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.342230 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.342535 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.342537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.342657 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.342738 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.342742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.343838 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.343840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.344361 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.344368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.344917 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.344920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.344976 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.345441 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.345451 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.347119 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 22:50:09.347357 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.342143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:36917 (local address 127.4.219.190:45213)
0925 22:50:09.342279 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.342284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.342300 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:09.342344 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.342349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.342415 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.342503 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.342876 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.343730 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.344469 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.344819 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.344923 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.347248 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:36917: BlockingRecv error: recv got EOF from 127.4.219.130:36917 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:50:09.347463 13986 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:50:09.445740 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:09.446003 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:09.446046 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:09.450019 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:09.450080 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:09.450138 13989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:09.455610 13989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/krb5kdc/oryx_127.4.219.131.keytab
--principal=oryx/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:37035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=45809
--tserver_master_addrs=127.4.219.190:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:09.456609 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:09.457567 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:09.469056 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.469058 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:09.469112 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:09.469283 13989 server_base.cc:1034] running on GCE node
I20260925 22:50:09.469761 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:09.470484 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:09.471686 13989 hybrid_clock.cc:648] HybridClock initialized: now 1790376609471667 us; error 31 us; skew 500 ppm
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376609, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:09.474805 13989 init.cc:377] Logged in from keytab as oryx/127.4.219.131@KRBTEST.COM (short username oryx)
I20260925 22:50:09.475736 13989 webserver.cc:460] Webserver started at http://127.4.219.131:45809/ using document root <none> and password file <none>
I20260925 22:50:09.476326 13989 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:09.476401 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:09.480394 13989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:50:09.482460 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.483727 13989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:09.483883 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "dcbc59460dc643c49d22eb4bdedb7baa"
format_stamp: "Formatted at 2026-09-25 22:50:08 on dist-test-slave-kb1g"
server_key: "ad96648b856beffdc8f14815bf48932f"
server_key_iv: "3656d7cde1c0213347984b15584ecf77"
server_key_version: "encryptionkey@0"
I20260925 22:50:09.484447 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:09.505508 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:09.508924 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:09.509156 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:09.509836 13989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:09.510684 13989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:09.510754 13989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.510831 13989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:09.510883 13989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:09.520586 13989 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:37035
I20260925 22:50:09.520656 14117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:37035 every 8 connection(s)
I20260925 22:50:09.521649 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:50:09.530279 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 13989
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:50:09.533043 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.523579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:38653 (local address 127.4.219.190:45213)
0925 22:50:09.523725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.523729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.524662 (+ 933us) server_negotiation.cc:407] Connection header received
0925 22:50:09.525970 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.525974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.526033 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.526112 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.527313 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.527840 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.529073 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.529299 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.529416 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.532923 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:38653: BlockingRecv error: recv got EOF from 127.4.219.131:38653 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:50:09.533828 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.524009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:38653)
0925 22:50:09.524462 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.524508 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.525741 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.526295 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.526302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.526669 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.527163 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.527177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.527969 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.527977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.528957 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.528965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.529451 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.529455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.529622 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.530226 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.530243 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.532819 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
W20260925 22:50:09.534322 14118 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:45213 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.219.190:45213: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.540366 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.534798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:41777)
0925 22:50:09.534914 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.534929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.534998 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.535198 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.535200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.535399 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.535511 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.535519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.536360 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.536367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.537138 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.537146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.537687 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.537689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.537769 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.538324 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.538335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.540270 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:50:09.540504 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.534868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:41777 (local address 127.4.219.190:45213)
0925 22:50:09.534992 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.534996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.535007 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:50:09.535058 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.535060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.535099 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.535178 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.535646 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.536248 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.537245 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.537568 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.537688 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.540389 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:41777: BlockingRecv error: recv got EOF from 127.4.219.131:41777 (error 108)
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":66}
I20260925 22:50:09.541671 13857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.531276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.4.219.190:45213)
0925 22:50:09.531409 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.531421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.531557 (+ 136us) server_negotiation.cc:407] Connection header received
0925 22:50:09.531713 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.531716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.531767 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.531856 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.532626 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.533193 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.534855 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.535220 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.535315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.536959 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:09.536964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:09.536979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:09.537007 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:09.539056 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.539586 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.539595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.539605 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.539764 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:09.540098 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:09.540101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:09.540102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:09.540590 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:09.540787 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:09.541324 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 22:50:09.541538 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:09 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:09.545703 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:09.540932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:44089)
0925 22:50:09.541018 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:09.541031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:09.541102 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:09.541359 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:09.541362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:09.541497 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:09.541562 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.541566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.542358 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.542361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.542975 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:09.542984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.543511 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.543514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:09.543600 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.544049 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:09.544060 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:09.545618 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 22:50:09.545845 13854 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:09.540999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44089 (local address 127.4.219.190:45213)
0925 22:50:09.541126 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:09.541132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:09.541147 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:09.541182 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:09.541185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:09.541219 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:09.541281 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:09.541667 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.542250 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.543088 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:09.543396 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:09.543510 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:09.545727 (+ 2217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:44089: BlockingRecv error: recv got EOF from 127.4.219.131:44089 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:50:09.545971 14118 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:10.166247 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:10.160893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:42317)
0925 22:50:10.161163 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:10.161177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:10.161265 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:10.161530 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:10.161532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:10.161702 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:10.161794 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.161799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.162582 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.162584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.163167 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.163175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.163659 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.163661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.163720 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.164299 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:10.164311 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:10.166123 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
W20260925 22:50:10.166482 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:10.160958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:42317 (local address 127.4.219.190:45213)
0925 22:50:10.161240 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:10.161244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:10.161259 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:10.161365 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:10.161368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:10.161414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:10.161497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:10.161897 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.162489 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.163298 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.163555 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.163649 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.166326 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:42317: BlockingRecv error: recv got EOF from 127.4.219.129:42317 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:10.353502 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:10.348034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:53773)
0925 22:50:10.348302 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:10.348315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:10.348417 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:10.348671 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:10.348672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:10.348846 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:10.348957 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.348962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.349662 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.349664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.350241 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.350249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.350644 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.350646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.350717 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.351218 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:10.351230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:10.353382 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
W20260925 22:50:10.353657 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:10.348167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:53773 (local address 127.4.219.190:45213)
0925 22:50:10.348308 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:10.348312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:10.348386 (+ 74us) server_negotiation.cc:407] Connection header received
0925 22:50:10.348495 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:10.348497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:10.348562 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:10.348638 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:10.349051 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.349562 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.350339 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.350551 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.350625 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.353532 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:53773: BlockingRecv error: recv got EOF from 127.4.219.130:53773 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:10 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:10.552670 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:10.546697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:53783)
0925 22:50:10.546965 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:10.546978 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:10.547081 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:10.547303 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:10.547305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:10.547479 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:10.547573 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.547579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.548641 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.548644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.549360 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:10.549369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.549893 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.549896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:10.549958 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.550526 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:10.550538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:10.552491 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
W20260925 22:50:10.552695 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:10.546774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:53783 (local address 127.4.219.190:45213)
0925 22:50:10.546933 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:10.546937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:10.547031 (+ 94us) server_negotiation.cc:407] Connection header received
0925 22:50:10.547148 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:10.547150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:10.547193 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:10.547272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:10.547679 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.548436 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.549506 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:10.549752 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:10.549865 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:10.552601 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:53783: BlockingRecv error: recv got EOF from 127.4.219.131:53783 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:11.172622 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:11.167116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:37953)
0925 22:50:11.167427 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:11.167440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:11.167567 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:11.167974 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:11.167977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:11.168160 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:11.168259 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.168265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.169012 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.169015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.169566 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.169573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.170001 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.170004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.170073 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.170554 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:11.170565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:11.172404 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
W20260925 22:50:11.172664 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:11.167152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:37953 (local address 127.4.219.190:45213)
0925 22:50:11.167422 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:11.167427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:11.167543 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:50:11.167788 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:11.167793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:11.167859 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:11.167940 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:11.168368 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.168915 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.169663 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.169883 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.169956 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.172523 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:37953: BlockingRecv error: recv got EOF from 127.4.219.129:37953 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:11.360179 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:11.354363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:55351)
0925 22:50:11.354628 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:11.354642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:11.354780 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:11.355040 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:11.355042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:11.355216 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:11.355317 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.355322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.356031 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.356035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.356882 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.356894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.357525 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.357529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.357621 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.358162 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:11.358176 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:11.360066 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
W20260925 22:50:11.360369 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:11.354454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:55351 (local address 127.4.219.190:45213)
0925 22:50:11.354634 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:11.354638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:11.354712 (+ 74us) server_negotiation.cc:407] Connection header received
0925 22:50:11.354871 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:11.354874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:11.354919 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:11.354989 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:11.355410 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.355919 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.357028 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.357400 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.357510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.360210 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:55351: BlockingRecv error: recv got EOF from 127.4.219.130:55351 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:11 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:11.563345 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:11.553395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:34385)
0925 22:50:11.553732 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:11.553747 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:11.553894 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:11.554177 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:11.554181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:11.554402 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:11.554538 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.554546 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.555472 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.555478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.556116 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:11.556125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.556622 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.556625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:11.556717 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.561094 (+ 4377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:11.561107 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:11.563121 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
W20260925 22:50:11.563390 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:11.553541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:34385 (local address 127.4.219.190:45213)
0925 22:50:11.553689 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:11.553694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:11.553826 (+ 132us) server_negotiation.cc:407] Connection header received
0925 22:50:11.554002 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:11.554006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:11.554060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:11.554150 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:11.554641 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.555365 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.556243 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:11.556475 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:11.556621 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:11.563281 (+ 6660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:34385: BlockingRecv error: recv got EOF from 127.4.219.131:34385 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:12.178819 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:12.173376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:55277)
0925 22:50:12.173674 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:12.173688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:12.173786 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:12.174044 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:12.174046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:12.174236 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:12.174332 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.174337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.175090 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.175093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.175653 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.175661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.176066 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.176068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.176128 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.176652 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:12.176664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:12.178637 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
W20260925 22:50:12.178906 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:12.173509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:55277 (local address 127.4.219.190:45213)
0925 22:50:12.173747 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:12.173750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:12.173764 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:12.173888 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:12.173890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:12.173934 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:12.174010 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:12.174440 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.174989 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.175749 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.175976 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.176050 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.178775 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:55277: BlockingRecv error: recv got EOF from 127.4.219.129:55277 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":57,"threads_started":1}
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:12.367164 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:12.361118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:58473)
0925 22:50:12.361404 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:12.361417 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:12.361520 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:12.361765 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:12.361767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:12.362004 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:12.362137 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.362144 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.363062 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.363066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.364009 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.364020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.364468 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.364471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.364555 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.365042 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:12.365054 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:12.367033 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":100,"threads_started":1}
W20260925 22:50:12.367349 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:12.361203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:58473 (local address 127.4.219.190:45213)
0925 22:50:12.361356 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:12.361361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:12.361487 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:50:12.361601 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:12.361603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:12.361649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:12.361730 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:12.362256 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.362920 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.364122 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.364350 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.364432 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.367184 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:58473: BlockingRecv error: recv got EOF from 127.4.219.130:58473 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:12 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:12.570230 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:12.564247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:47361)
0925 22:50:12.564501 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:12.564516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:12.564654 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:12.564969 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:12.564972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:12.565181 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:12.565295 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.565302 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.566088 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.566091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.566650 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:12.566658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.567352 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.567355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:12.567425 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.567955 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:12.567966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:12.570033 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
W20260925 22:50:12.570268 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:12.564364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:47361 (local address 127.4.219.190:45213)
0925 22:50:12.564493 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:12.564499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:12.564603 (+ 104us) server_negotiation.cc:407] Connection header received
0925 22:50:12.564761 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:12.564764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:12.564830 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:12.564924 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:12.565384 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.565981 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.566925 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:12.567182 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:12.567279 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:12.570159 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:47361: BlockingRecv error: recv got EOF from 127.4.219.131:47361 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:13.186229 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:13.179901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:37049)
0925 22:50:13.180230 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:13.180246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:13.180346 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:13.180629 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:13.180631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:13.180825 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:13.180925 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.180931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.181858 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.181861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.182446 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.182456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.183032 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.183035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.183102 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.183682 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:13.183694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:13.186106 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":115,"threads_started":1}
W20260925 22:50:13.186499 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:13.180015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:37049 (local address 127.4.219.190:45213)
0925 22:50:13.180275 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:13.180278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:13.180299 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:50:13.180438 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:13.180441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:13.180485 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:13.180594 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:13.181092 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.181715 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.182562 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.182893 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.182987 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.186240 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:37049: BlockingRecv error: recv got EOF from 127.4.219.129:37049 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:13.373436 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:13.367982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:48847)
0925 22:50:13.368251 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:13.368265 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:13.368377 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:13.368612 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:13.368614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:13.368801 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:13.368900 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.368905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.369689 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.369692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.370280 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.370290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.370723 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.370726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.370787 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.371411 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:13.371423 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:13.373318 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260925 22:50:13.373646 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:13.368037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:48847 (local address 127.4.219.190:45213)
0925 22:50:13.368184 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:13.368188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:13.368338 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:50:13.368458 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:13.368460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:13.368504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:13.368602 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:13.369003 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.369526 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.370389 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.370600 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.370674 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.373506 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:48847: BlockingRecv error: recv got EOF from 127.4.219.130:48847 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:13 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:13.576362 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:13.571058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:51307)
0925 22:50:13.571348 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:13.571361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:13.571458 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:13.571684 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:13.571686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:13.571861 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:13.571959 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.571964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.572686 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.572689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.573239 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:13.573247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.573649 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.573651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:13.573708 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.574192 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:13.574203 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:13.576175 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":113,"threads_started":1}
W20260925 22:50:13.576421 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:13.571157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:51307 (local address 127.4.219.190:45213)
0925 22:50:13.571294 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:13.571297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:13.571430 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:50:13.571535 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:13.571538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:13.571583 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:13.571654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:13.572055 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.572591 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.573340 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:13.573557 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:13.573632 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:13.576279 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:51307: BlockingRecv error: recv got EOF from 127.4.219.131:51307 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:14.193372 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:14.187110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:39649)
0925 22:50:14.187444 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:14.187459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:14.187550 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:14.187853 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:14.187855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:14.188035 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:14.188127 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.188132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.189295 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.189298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.189968 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.189977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.190409 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.190411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.190498 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.191178 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:14.191189 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:14.193196 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
W20260925 22:50:14.193450 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:14.187266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:39649 (local address 127.4.219.190:45213)
0925 22:50:14.187524 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:14.187528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:14.187543 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:14.187643 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:14.187646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:14.187690 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:14.187801 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:14.188340 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.189186 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.190076 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.190313 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.190395 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.193325 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:39649: BlockingRecv error: recv got EOF from 127.4.219.129:39649 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:14.379995 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:14.374396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:47887)
0925 22:50:14.374689 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:14.374703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:14.374788 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:14.375019 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:14.375021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:14.375201 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:14.375309 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.375314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.376062 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.376065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.376666 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.376674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.377123 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.377126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.377183 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.377720 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:14.377732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:14.379822 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
W20260925 22:50:14.380054 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:14.374580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:47887 (local address 127.4.219.190:45213)
0925 22:50:14.374751 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:14.374754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:14.374770 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:14.374860 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:14.374862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:14.374905 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:14.374982 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:14.375427 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.375968 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.376784 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.377021 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.377109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.379947 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:47887: BlockingRecv error: recv got EOF from 127.4.219.130:47887 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:14 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:14.583441 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:14.577499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:48873)
0925 22:50:14.577771 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:14.577784 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:14.577885 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:14.578236 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:14.578239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:14.578435 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:14.578533 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.578539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.579542 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.579546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.580213 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:14.580225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.580794 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.580796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:14.580858 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.581340 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:14.581351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:14.583264 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
W20260925 22:50:14.583503 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:14.577589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:48873 (local address 127.4.219.190:45213)
0925 22:50:14.577787 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:14.577790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:14.577866 (+ 76us) server_negotiation.cc:407] Connection header received
0925 22:50:14.578023 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:14.578027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:14.578077 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:14.578164 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:14.578681 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.579401 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.580356 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:14.580686 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:14.580786 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:14.583392 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:48873: BlockingRecv error: recv got EOF from 127.4.219.131:48873 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:15.200033 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:15.194396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:42133)
0925 22:50:15.194662 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:15.194676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:15.194766 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:15.195001 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:15.195003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:15.195182 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:15.195283 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.195288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.196042 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.196045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.196677 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.196688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.197111 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.197114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.197172 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.197722 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:15.197733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:15.199858 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260925 22:50:15.200104 14143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:15.194458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:42133 (local address 127.4.219.190:45213)
0925 22:50:15.194716 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:15.194720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:15.194735 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:15.194859 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:15.194861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:15.194905 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:15.194989 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:15.195401 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.195946 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.196790 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.197010 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.197101 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.199967 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:42133: BlockingRecv error: recv got EOF from 127.4.219.129:42133 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":145,"threads_started":1}
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:15.386552 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:15.380838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:38439)
0925 22:50:15.381114 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:15.381132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:15.381243 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:15.381529 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:15.381532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:15.381747 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:15.381837 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.381843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.382649 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.382654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.383213 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.383220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.383670 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.383674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.383761 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.384303 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:15.384316 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:15.386434 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260925 22:50:15.386806 14143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:15.381030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38439 (local address 127.4.219.190:45213)
0925 22:50:15.381199 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:15.381204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:15.381219 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:50:15.381358 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:15.381362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:15.381415 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:15.381504 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:15.381955 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.382513 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.383331 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.383556 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.383669 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.386598 (+ 2929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:38439: BlockingRecv error: recv got EOF from 127.4.219.130:38439 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:15 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:15.589924 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:15.584306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:43819)
0925 22:50:15.584582 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:15.584596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:15.584698 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:15.584909 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:15.584911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:15.585086 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:15.585176 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.585181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.585968 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.585971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.586519 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:15.586526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.586972 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.586974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:15.587031 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.587658 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:15.587670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:15.589737 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
W20260925 22:50:15.589982 14143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:15.584414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:43819 (local address 127.4.219.190:45213)
0925 22:50:15.584603 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:15.584608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:15.584671 (+ 63us) server_negotiation.cc:407] Connection header received
0925 22:50:15.584780 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:15.584783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:15.584830 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:15.584898 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:15.585305 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.585870 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.586660 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:15.586875 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:15.586955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:15.589874 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:43819: BlockingRecv error: recv got EOF from 127.4.219.131:43819 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:16.206728 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:16.200944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:51351)
0925 22:50:16.201255 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:16.201275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:16.201373 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:16.201680 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:16.201682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:16.201860 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:16.201950 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.201955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.202748 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.202752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.203337 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.203347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.203874 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.203877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.203960 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.204515 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:16.204570 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:16.206506 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
W20260925 22:50:16.206769 14147 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:16.201010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:51351 (local address 127.4.219.190:45213)
0925 22:50:16.201281 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:16.201285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:16.201330 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:50:16.201503 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:16.201506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:16.201555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:16.201647 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:16.202061 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.202636 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.203471 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.203737 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.203831 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.206641 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:51351: BlockingRecv error: recv got EOF from 127.4.219.129:51351 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:16.393433 14148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:16.387413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:33061)
0925 22:50:16.387726 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:16.387762 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:16.387885 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:16.388196 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:16.388199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:16.388384 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:16.388575 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.388586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.389461 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.389463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.390033 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.390041 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.390587 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.390590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.390663 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.391164 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:16.391176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:16.393291 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260925 22:50:16.393546 14147 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:16.387499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:33061 (local address 127.4.219.190:45213)
0925 22:50:16.387702 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:16.387707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:16.387874 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:50:16.387989 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:16.387991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:16.388039 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:16.388123 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:16.388730 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.389341 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.390192 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.390478 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.390559 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.393426 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:33061: BlockingRecv error: recv got EOF from 127.4.219.130:33061 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:16 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:16.597062 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:16.590783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:52159)
0925 22:50:16.591139 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:16.591153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:16.591304 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:16.591594 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:16.591597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:16.591859 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:16.591955 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.591961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.592837 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.592844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.593502 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:16.593513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.594038 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.594042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:16.594135 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.594699 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:16.594711 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:16.596886 (+ 2175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
W20260925 22:50:16.597182 14147 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:16.590855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:52159 (local address 127.4.219.190:45213)
0925 22:50:16.591035 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:16.591040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:16.591240 (+ 200us) server_negotiation.cc:407] Connection header received
0925 22:50:16.591424 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:16.591428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:16.591481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:16.591572 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:16.592086 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.592729 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.593632 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:16.593919 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:16.594037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:16.597022 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:52159: BlockingRecv error: recv got EOF from 127.4.219.131:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:17.216310 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:17.207600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:33007)
0925 22:50:17.207901 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:17.207916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:17.208004 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:17.208307 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:17.208309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:17.208476 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:17.208599 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.208605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.209321 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.209323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.209878 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.209886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.210399 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.210403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.210467 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.214347 (+ 3880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:17.214359 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:17.216182 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":90,"threads_started":1}
W20260925 22:50:17.216507 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:17.207853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:33007 (local address 127.4.219.190:45213)
0925 22:50:17.208081 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:17.208085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:17.208101 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:17.208141 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:17.208143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:17.208183 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:17.208271 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:17.208694 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.209222 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.209998 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.210272 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.210385 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.216329 (+ 5944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:33007: BlockingRecv error: recv got EOF from 127.4.219.129:33007 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:17.400255 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:17.394480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:49831)
0925 22:50:17.394818 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:17.394832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:17.394934 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:17.395200 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:17.395203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:17.395378 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:17.395472 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.395477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.396433 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.396435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.397083 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.397093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.397540 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.397543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.397600 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.398226 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:17.398240 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:17.400145 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":154,"threads_started":1}
W20260925 22:50:17.400465 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:17.394553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49831 (local address 127.4.219.190:45213)
0925 22:50:17.394714 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:17.394719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:17.394935 (+ 216us) server_negotiation.cc:407] Connection header received
0925 22:50:17.395006 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:17.395008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:17.395058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:17.395142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:17.395605 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.396337 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.397193 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.397442 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.397528 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.400335 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:49831: BlockingRecv error: recv got EOF from 127.4.219.130:49831 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:17 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:17.602905 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:17.597877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:59333)
0925 22:50:17.598150 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:17.598162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:17.598258 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:17.598523 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:17.598526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:17.598710 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:17.598826 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.598832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.599499 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.599501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.600100 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:17.600108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.600493 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.600495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:17.600616 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.601093 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:17.601104 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:17.602793 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":114,"threads_started":1}
W20260925 22:50:17.603039 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:17.597911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:59333 (local address 127.4.219.190:45213)
0925 22:50:17.598049 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:17.598053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:17.598225 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:50:17.598322 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:17.598324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:17.598369 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:17.598485 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:17.598908 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.599412 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.600197 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:17.600403 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:17.600475 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:17.602923 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:59333: BlockingRecv error: recv got EOF from 127.4.219.131:59333 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:18.222601 14154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.217081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:54363)
0925 22:50:18.217380 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.217397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.217545 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.217822 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.217824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.218026 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.218115 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.218120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.218876 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.218881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.219434 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.219441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.219906 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.219910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.219994 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.220484 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.220495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.222366 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
W20260925 22:50:18.222633 14155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.217122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:54363 (local address 127.4.219.190:45213)
0925 22:50:18.217388 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.217394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.217482 (+ 88us) server_negotiation.cc:407] Connection header received
0925 22:50:18.217641 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.217645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.217692 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.217799 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.218238 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.218765 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.219553 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.219798 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.219904 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.222494 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:54363: BlockingRecv error: recv got EOF from 127.4.219.129:54363 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:18.406033 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.400917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:44603)
0925 22:50:18.401169 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.401182 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.401283 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.401512 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.401514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.401687 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.401793 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.401799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.402492 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.402494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.403056 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.403063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.403459 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.403461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.403524 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.404019 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.404030 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.405920 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
W20260925 22:50:18.406211 14155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.401004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:44603 (local address 127.4.219.190:45213)
0925 22:50:18.401134 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.401137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.401246 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:50:18.401363 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.401365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.401408 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.401477 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.401891 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.402407 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.403152 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.403364 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.403437 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.406055 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:44603: BlockingRecv error: recv got EOF from 127.4.219.130:44603 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:18 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:18.609277 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:18.603759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:51469)
0925 22:50:18.604039 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:18.604052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:18.604154 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:18.604410 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:18.604412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:18.604617 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:18.604710 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.604715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.605398 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.605400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.605932 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:18.605938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.606321 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.606324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:18.606398 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.606994 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:18.607006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:18.609140 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
W20260925 22:50:18.609359 14155 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:18.603868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:51469 (local address 127.4.219.190:45213)
0925 22:50:18.604063 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:18.604067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:18.604135 (+ 68us) server_negotiation.cc:407] Connection header received
0925 22:50:18.604259 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:18.604261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:18.604308 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:18.604375 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:18.604807 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.605311 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.606025 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:18.606231 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:18.606303 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:18.609273 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:51469: BlockingRecv error: recv got EOF from 127.4.219.131:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:19.229432 14158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.223454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:42257)
0925 22:50:19.223756 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:19.223771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:19.223870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:19.224131 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:19.224133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:19.224316 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:19.224445 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.224450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.225379 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.225383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.226168 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.226179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.226659 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.226663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.226748 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.227278 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:19.227290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:19.229307 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":120,"threads_started":1}
W20260925 22:50:19.229839 14159 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:19.223549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:42257 (local address 127.4.219.190:45213)
0925 22:50:19.223784 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.223788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.223826 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:50:19.223951 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.223954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.223996 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.224077 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:19.224607 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.225253 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.226287 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.226526 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.226650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.229743 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:42257: BlockingRecv error: recv got EOF from 127.4.219.129:42257 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:19.411794 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.406749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:57129)
0925 22:50:19.406995 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:19.407008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:19.407098 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:19.407324 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:19.407326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:19.407497 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:19.407588 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.407593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.408288 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.408290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.408906 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.408915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.409313 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.409315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.409375 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.409844 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:19.409855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:19.411667 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260925 22:50:19.411926 14159 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:19.406832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:57129 (local address 127.4.219.190:45213)
0925 22:50:19.407014 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.407017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.407057 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:50:19.407181 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.407184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.407226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.407293 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:19.407684 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.408200 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.409010 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.409223 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.409301 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.411801 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:57129: BlockingRecv error: recv got EOF from 127.4.219.130:57129 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:19 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:19.615808 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:19.610222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:32919)
0925 22:50:19.610556 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:19.610573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:19.610689 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:19.610981 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:19.610984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:19.611187 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:19.611291 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.611297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.612058 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.612065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.612713 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:19.612722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.613180 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.613183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:19.613277 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.613832 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:19.613844 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:19.615684 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
W20260925 22:50:19.615944 14159 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:19.610447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:32919 (local address 127.4.219.190:45213)
0925 22:50:19.610608 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:19.610615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:19.610632 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:50:19.610804 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:19.610807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:19.610861 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:19.610955 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:19.611418 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.611946 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.612841 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:19.613069 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:19.613180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:19.615827 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:32919: BlockingRecv error: recv got EOF from 127.4.219.131:32919 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:20.236152 14162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:20.230447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:41929)
0925 22:50:20.230768 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:20.230782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:20.230883 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:20.231134 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:20.231137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:20.231312 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:20.231451 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.231458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.232488 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.232491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.233271 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.233281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.233688 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.233691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.233751 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.234231 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:20.234243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:20.236030 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":91,"threads_started":1}
W20260925 22:50:20.236304 14163 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:20.230592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:41929 (local address 127.4.219.190:45213)
0925 22:50:20.230858 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:20.230861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:20.230878 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:50:20.230975 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:20.230977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:20.231021 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:20.231099 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:20.231560 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.232353 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.233379 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.233593 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.233670 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.236168 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:41929: BlockingRecv error: recv got EOF from 127.4.219.129:41929 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:50:20.420292 14163 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:20.412818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:56721 (local address 127.4.219.190:45213)
0925 22:50:20.413065 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:20.413071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:20.413195 (+ 124us) server_negotiation.cc:407] Connection header received
0925 22:50:20.413369 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:20.413375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:20.413438 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:20.413539 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:20.414040 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.414618 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.415789 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.416223 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.416363 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.420193 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:56721: BlockingRecv error: recv got EOF from 127.4.219.130:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 22:50:20.420385 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:20.412761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:56721)
0925 22:50:20.413081 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:20.413100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:20.413262 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:20.413600 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:20.413603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:20.413816 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:20.413910 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.413915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.414738 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.414746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.415630 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.415640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.416374 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.416378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.416508 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.417294 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:20.417310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:20.420059 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:20 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:20.622332 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:20.616700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:55967)
0925 22:50:20.616954 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:20.616973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:20.617084 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:20.617324 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:20.617327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:20.617499 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:20.617598 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.617603 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.618333 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.618336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.619047 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:20.619079 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.619563 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.619567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:20.619655 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.620278 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:20.620290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:20.622167 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260925 22:50:20.622391 14163 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:20.616763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:55967 (local address 127.4.219.190:45213)
0925 22:50:20.616905 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:20.616915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:20.617053 (+ 138us) server_negotiation.cc:407] Connection header received
0925 22:50:20.617169 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:20.617176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:20.617223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:20.617282 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:20.617700 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.618225 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.619197 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:20.619445 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:20.619550 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:20.622287 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:55967: BlockingRecv error: recv got EOF from 127.4.219.131:55967 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:21.242252 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.237020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:47959)
0925 22:50:21.237285 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.237299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.237387 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.237638 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.237640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.237814 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.237906 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.237911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.238595 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.238598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.239160 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.239168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.239561 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.239563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.239624 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.240093 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.240105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.242079 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260925 22:50:21.242321 14167 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.237078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:47959 (local address 127.4.219.190:45213)
0925 22:50:21.237325 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.237329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.237343 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:21.237477 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.237480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.237524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.237600 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.237997 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.238507 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.239253 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.239468 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.239541 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.242200 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:47959: BlockingRecv error: recv got EOF from 127.4.219.129:47959 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":57,"threads_started":1}
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:21.426556 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.421242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:40783)
0925 22:50:21.421532 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.421544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.421636 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.421862 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.421864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.422035 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.422127 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.422131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.422966 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.422969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.423530 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.423538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.423975 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.423977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.424052 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.424708 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.424732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.426437 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":129,"threads_started":1}
W20260925 22:50:21.426798 14167 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.421288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:40783 (local address 127.4.219.190:45213)
0925 22:50:21.421442 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.421447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.421593 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:50:21.421713 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.421716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.421763 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.421833 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.422231 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.422842 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.423636 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.423877 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.423959 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.426690 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:40783: BlockingRecv error: recv got EOF from 127.4.219.130:40783 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:21 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:21.628427 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:21.623131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:34901)
0925 22:50:21.623413 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:21.623434 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:21.623536 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:21.623736 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:21.623738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:21.623927 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:21.624036 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.624042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.624780 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.624784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.625345 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:21.625353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.625910 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.625914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:21.625987 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.626471 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:21.626483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:21.628313 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260925 22:50:21.628640 14167 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:21.623191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:34901 (local address 127.4.219.190:45213)
0925 22:50:21.623324 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:21.623328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:21.623501 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:50:21.623605 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:21.623606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:21.623657 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:21.623721 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:21.624131 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.624671 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.625444 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:21.625647 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:21.625728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:21.628435 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:34901: BlockingRecv error: recv got EOF from 127.4.219.131:34901 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:22.253971 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.243120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:37177)
0925 22:50:22.243417 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:22.243434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:22.243556 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:22.243828 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:22.243830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:22.244032 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:22.244129 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.244134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.245100 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.245105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.245788 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.245799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.246285 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.246289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.246377 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.251718 (+ 5341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:22.251732 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:22.253844 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":87,"threads_started":1}
W20260925 22:50:22.254190 14171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:22.243216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:37177 (local address 127.4.219.190:45213)
0925 22:50:22.243454 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.243458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.243524 (+ 66us) server_negotiation.cc:407] Connection header received
0925 22:50:22.243660 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.243662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.243706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.243785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:22.244234 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.244964 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.245924 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.246164 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.246281 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.254043 (+ 7762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:37177: BlockingRecv error: recv got EOF from 127.4.219.129:37177 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":57,"threads_started":1}
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:22.432941 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.427254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:38143)
0925 22:50:22.427571 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:22.427587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:22.427700 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:22.427977 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:22.427980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:22.428196 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:22.428283 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.428288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.429157 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.429163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.429731 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.429739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.430212 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.430215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.430298 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.430884 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:22.430896 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:22.432770 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
W20260925 22:50:22.432996 14171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:22.427411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38143 (local address 127.4.219.190:45213)
0925 22:50:22.427561 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.427565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.427643 (+ 78us) server_negotiation.cc:407] Connection header received
0925 22:50:22.427809 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.427812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.427865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.427953 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:22.428397 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.429047 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.429869 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.430100 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.430201 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.432891 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:38143: BlockingRecv error: recv got EOF from 127.4.219.130:38143 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:22 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:22.635035 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:22.629198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:41109)
0925 22:50:22.629466 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:22.629483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:22.629601 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:22.629908 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:22.629912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:22.630102 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:22.630199 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.630204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.631040 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.631043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.631618 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:22.631625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.632235 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.632237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:22.632298 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.632889 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:22.632901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:22.634891 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260925 22:50:22.635157 14171 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:22.629253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:41109 (local address 127.4.219.190:45213)
0925 22:50:22.629413 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:22.629418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:22.629566 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:50:22.629700 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:22.629703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:22.629765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:22.629872 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:22.630288 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.630933 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.631893 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:22.632130 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:22.632221 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:22.635014 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:41109: BlockingRecv error: recv got EOF from 127.4.219.131:41109 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:23.259914 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:23.254728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:56901)
0925 22:50:23.254996 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:23.255010 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:23.255098 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:23.255338 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:23.255340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:23.255505 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:23.255591 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.255595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.256342 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.256344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.256979 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.256987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.257394 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.257396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.257456 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.257942 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:23.257952 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:23.259736 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
W20260925 22:50:23.259990 14175 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:23.254790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:56901 (local address 127.4.219.190:45213)
0925 22:50:23.255096 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:23.255102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:23.255118 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:50:23.255160 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:23.255163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:23.255212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:23.255307 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:23.255700 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.256252 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.257082 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.257302 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.257380 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.259853 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:56901: BlockingRecv error: recv got EOF from 127.4.219.129:56901 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":57,"threads_started":1}
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:23.439429 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:23.433896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:51891)
0925 22:50:23.434205 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:23.434219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:23.434324 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:23.434567 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:23.434569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:23.434745 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:23.434831 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.434836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.435629 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.435631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.436179 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.436186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.436680 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.436683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.436744 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.437253 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:23.437264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:23.439243 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
W20260925 22:50:23.439576 14175 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:23.434070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:51891 (local address 127.4.219.190:45213)
0925 22:50:23.434228 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:23.434232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:23.434275 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:50:23.434409 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:23.434411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:23.434460 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:23.434530 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:23.434963 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.435531 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.436290 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.436576 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.436655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.439440 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:51891: BlockingRecv error: recv got EOF from 127.4.219.130:51891 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:23 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.131@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:23.641780 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:23.635833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.131:37137)
0925 22:50:23.636177 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:23.636197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:23.636328 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:23.636640 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:23.636643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:23.636849 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:23.636940 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.636946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.637737 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.637743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.638463 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:23.638474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.638957 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.638961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:23.639048 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.639690 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:23.639702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:23.641582 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
W20260925 22:50:23.641821 14175 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:23.635972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:37137 (local address 127.4.219.190:45213)
0925 22:50:23.636138 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:23.636142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:23.636269 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:50:23.636425 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:23.636429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:23.636484 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:23.636615 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:23.637057 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.637609 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.638582 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:23.638857 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:23.638956 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:23.641715 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.131:37137: BlockingRecv error: recv got EOF from 127.4.219.131:37137 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 22:50:24 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:24 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.129@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:24.266220 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:24.260890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.129:52505)
0925 22:50:24.261166 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:24.261179 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:24.261308 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:24.261628 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:24.261630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:24.261801 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:24.261890 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.261895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.262706 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.262708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.263266 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.263273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.263714 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.263716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.263774 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:24.264256 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:24.264266 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:24.266096 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
W20260925 22:50:24.266500 14179 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:24.260950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52505 (local address 127.4.219.190:45213)
0925 22:50:24.261174 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:24.261179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:24.261269 (+ 90us) server_negotiation.cc:407] Connection header received
0925 22:50:24.261430 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:24.261433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:24.261483 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:24.261574 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:24.261990 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.262612 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.263381 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.263615 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.263700 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:24.266266 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.129:52505: BlockingRecv error: recv got EOF from 127.4.219.129:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":57,"threads_started":1}
Sep 25 22:50:24 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:50:24 dist-test-slave-kb1g krb5kdc[13121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.219.130@KRBTEST.COM for oryx/127.4.219.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:50:24.446327 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:24.440508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45213 (local address 127.4.219.130:57421)
0925 22:50:24.440836 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:24.440853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:24.440952 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:24.441175 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:24.441177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:24.441479 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:24.441576 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.441582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.442333 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.442336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.442944 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.442953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.443385 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.443387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.443488 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:24.444105 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:24.444117 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:24.446215 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.219.190:45213: Server oryx/127.4.219.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
W20260925 22:50:24.446436 14179 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:24.440675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:57421 (local address 127.4.219.190:45213)
0925 22:50:24.440813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:24.440817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:24.440922 (+ 105us) server_negotiation.cc:407] Connection header received
0925 22:50:24.441023 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:24.441026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:24.441068 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:24.441140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:24.441690 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.442213 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.443069 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.443284 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.443364 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:24.446341 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.219.130:57421: BlockingRecv error: recv got EOF from 127.4.219.130:57421 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:50:24.537854 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13704
I20260925 22:50:24.543466 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13856
I20260925 22:50:24.548640 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13989
I20260925 22:50:24.553977 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 13634
2026-09-25T22:50:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18015 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:24Z Disabled control of system clock
I20260925 22:50:24.584617 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:33403
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:33403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:50:24.699604 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:24.699832 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:24.699864 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:24.703873 14184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:24.703929 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:24.703945 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:24.703961 14184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:24.703976 14184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:24.708673 14184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:33403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:33403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:24.709679 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:24.710937 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:24.719111 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:24.719121 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:24.719121 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:24.719267 14184 server_base.cc:1034] running on GCE node
I20260925 22:50:24.720005 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:24.721179 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:24.722395 14184 hybrid_clock.cc:648] HybridClock initialized: now 1790376624722376 us; error 28 us; skew 500 ppm
I20260925 22:50:24.724248 14184 webserver.cc:460] Webserver started at http://127.4.219.190:40213/ using document root <none> and password file <none>
I20260925 22:50:24.724893 14184 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:24.724992 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:24.725219 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:24.727217 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "a585be09ba414bfca1dfd6b511980c75"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "afe84b3e724bec5a9b195a5aa181e149"
server_key_iv: "27dbf4f72f5a30aa6c6e1ef7e59503ad"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.727762 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "a585be09ba414bfca1dfd6b511980c75"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "afe84b3e724bec5a9b195a5aa181e149"
server_key_iv: "27dbf4f72f5a30aa6c6e1ef7e59503ad"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.731395 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:50:24.734266 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:24.736052 14184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:50:24.736202 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "a585be09ba414bfca1dfd6b511980c75"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "afe84b3e724bec5a9b195a5aa181e149"
server_key_iv: "27dbf4f72f5a30aa6c6e1ef7e59503ad"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.736334 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:24.752419 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:24.753166 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:24.753398 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:24.760363 14184 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:33403
I20260925 22:50:24.760383 14250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:33403 every 8 connection(s)
I20260925 22:50:24.761451 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:24.764736 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:24.770081 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14184
I20260925 22:50:24.770159 14251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75: Bootstrap starting.
I20260925 22:50:24.770202 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:50:24.770496 4974 external_mini_cluster.cc:1428] Setting key 85c261145861c670b13370708babcb63
I20260925 22:50:24.772739 14251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:24.773456 14251 log.cc:824] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:24.775202 14251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75: No bootstrap required, opened a new log
I20260925 22:50:24.778584 14251 raft_consensus.cc:348] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } }
I20260925 22:50:24.778855 14251 raft_consensus.cc:374] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:24.778950 14251 raft_consensus.cc:729] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a585be09ba414bfca1dfd6b511980c75, State: Initialized, Role: FOLLOWER
I20260925 22:50:24.779415 14251 consensus_queue.cc:260] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } }
I20260925 22:50:24.779573 14251 raft_consensus.cc:388] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:24.779661 14251 raft_consensus.cc:482] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:24.779757 14251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:24.779807 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:24.771885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.4.219.190:33403)
0925 22:50:24.772504 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:24.772519 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:24.772613 (+ 94us) server_negotiation.cc:407] Connection header received
0925 22:50:24.773277 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:24.773295 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:24.773730 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:24.774073 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:24.774486 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.775490 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.776384 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.776931 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.777098 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:24.777965 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:24.777977 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:24.777990 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:24.778350 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:24.778405 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:24.778423 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:24.778529 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:24.778649 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 22:50:24.778973 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260925 22:50:24.780808 14251 raft_consensus.cc:504] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } }
I20260925 22:50:24.781129 14251 leader_election.cc:302] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a585be09ba414bfca1dfd6b511980c75; no voters:
I20260925 22:50:24.781409 14251 leader_election.cc:288] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:24.781507 14256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:24.781708 14256 raft_consensus.cc:686] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [term 1 LEADER]: Becoming Leader. State: Replica: a585be09ba414bfca1dfd6b511980c75, State: Running, Role: LEADER
I20260925 22:50:24.782120 14256 consensus_queue.cc:237] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } }
I20260925 22:50:24.782595 14251 sys_catalog.cc:564] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:24.783264 14256 sys_catalog.cc:455] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a585be09ba414bfca1dfd6b511980c75. Latest consensus state: current_term: 1 leader_uuid: "a585be09ba414bfca1dfd6b511980c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } } }
I20260925 22:50:24.783383 14256 sys_catalog.cc:458] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:24.783797 14257 sys_catalog.cc:455] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a585be09ba414bfca1dfd6b511980c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a585be09ba414bfca1dfd6b511980c75" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33403 } } }
I20260925 22:50:24.783969 14257 sys_catalog.cc:458] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:24.783808 14264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:24.787140 14264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:24.792497 14264 catalog_manager.cc:1347] Generated new cluster ID: 1cd3c47a9b564c5bafc076bed8cfebd0
I20260925 22:50:24.792578 14264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:24.799650 14264 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:24.800840 14264 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:24.810345 14264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a585be09ba414bfca1dfd6b511980c75: Generated new TSK 0
I20260925 22:50:24.811158 14264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:24.816689 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--builtin_ntp_servers=127.4.219.148:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:50:24.929312 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:24.929562 14275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:24.929598 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:24.933642 14275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:24.933687 14275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:24.933790 14275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:24.938495 14275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:24.939592 14275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:24.941089 14275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:24.951004 14283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:24.951004 14280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:24.951053 14275 server_base.cc:1034] running on GCE node
W20260925 22:50:24.951022 14281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:24.951699 14275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:24.952363 14275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:24.953534 14275 hybrid_clock.cc:648] HybridClock initialized: now 1790376624953516 us; error 34 us; skew 500 ppm
I20260925 22:50:24.955397 14275 webserver.cc:460] Webserver started at http://127.4.219.129:41543/ using document root <none> and password file <none>
I20260925 22:50:24.956034 14275 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:24.956099 14275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:24.956357 14275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:24.958489 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "8eeea123ac454c568b15eab4d93bcdae"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "bc434841119f1d7c3729146f7b6ec658"
server_key_iv: "0ba7aa564bf26b87030e338d132ed7ca"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.959054 14275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "8eeea123ac454c568b15eab4d93bcdae"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "bc434841119f1d7c3729146f7b6ec658"
server_key_iv: "0ba7aa564bf26b87030e338d132ed7ca"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.962783 14275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:50:24.965315 14289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:24.966508 14275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:24.966642 14275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "8eeea123ac454c568b15eab4d93bcdae"
format_stamp: "Formatted at 2026-09-25 22:50:24 on dist-test-slave-kb1g"
server_key: "bc434841119f1d7c3729146f7b6ec658"
server_key_iv: "0ba7aa564bf26b87030e338d132ed7ca"
server_key_version: "encryptionkey@0"
I20260925 22:50:24.966745 14275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:24.978422 14275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:24.979015 14275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:24.979156 14275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:24.979636 14275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:24.980464 14275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:24.980508 14275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:24.980614 14275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:24.980657 14275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:24.989166 14275 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:38387
I20260925 22:50:24.989207 14402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:38387 every 8 connection(s)
I20260925 22:50:24.990058 14275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:24.994550 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14275
I20260925 22:50:24.994670 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:50:24.995011 4974 external_mini_cluster.cc:1428] Setting key 9669626b3bb537561d033e455144ec72
I20260925 22:50:24.997139 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--builtin_ntp_servers=127.4.219.148:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:50:25.000912 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:24.991868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:47949 (local address 127.4.219.190:33403)
0925 22:50:24.992026 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:24.992029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:24.993186 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 22:50:24.994819 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:24.994822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:24.994870 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:24.994950 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:24.996044 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.996792 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.998584 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.998981 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.999100 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.000328 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:25.000332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:25.000334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:25.000354 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:25.000373 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:25.000381 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:25.000470 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:25.000677 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:50:25.000774 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:50:25.001857 14405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:24.992299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33403 (local address 127.4.219.129:47949)
0925 22:50:24.993038 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:24.993080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:24.994557 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:24.995161 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:24.995171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:24.995280 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:24.995907 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.995922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.996944 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.996950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.998443 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:24.998458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:24.999137 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:24.999140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:24.999350 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.000068 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:25.000069 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:25.000109 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:25.000173 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:25.000473 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:25.000484 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:25.000491 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:50:25.000677 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:50:25.000989 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260925 22:50:25.003054 14403 heartbeater.cc:347] Connected to a master server at 127.4.219.190:33403
I20260925 22:50:25.003311 14403 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:25.003831 14403 heartbeater.cc:511] Master 127.4.219.190:33403 requested a full tablet report, sending...
I20260925 22:50:25.005474 14215 ts_manager.cc:194] Registered new tserver with Master: 8eeea123ac454c568b15eab4d93bcdae (127.4.219.129:38387)
I20260925 22:50:25.006613 14215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.129:47949
W20260925 22:50:25.111816 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:25.112150 14406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:25.112223 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:25.116384 14406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:25.116485 14406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:25.116597 14406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:25.121467 14406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:25.122709 14406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:25.124221 14406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:25.133411 14411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.133419 14414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.133466 14412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.133695 14406 server_base.cc:1034] running on GCE node
I20260925 22:50:25.134159 14406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:25.134831 14406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:25.136015 14406 hybrid_clock.cc:648] HybridClock initialized: now 1790376625136007 us; error 28 us; skew 500 ppm
I20260925 22:50:25.137858 14406 webserver.cc:460] Webserver started at http://127.4.219.130:43075/ using document root <none> and password file <none>
I20260925 22:50:25.138424 14406 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:25.138512 14406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:25.138748 14406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:25.140861 14406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "5f735c69a61346378eb84f92cd1bf7de"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "4047e98c509dfb95017bf59ce1040cab"
server_key_iv: "3cf592fb4ec40d0315fc0cac7caea2c4"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.141438 14406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "5f735c69a61346378eb84f92cd1bf7de"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "4047e98c509dfb95017bf59ce1040cab"
server_key_iv: "3cf592fb4ec40d0315fc0cac7caea2c4"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.145282 14406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:50:25.147745 14420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.148986 14406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:25.149113 14406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "5f735c69a61346378eb84f92cd1bf7de"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "4047e98c509dfb95017bf59ce1040cab"
server_key_iv: "3cf592fb4ec40d0315fc0cac7caea2c4"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.149223 14406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:25.165407 14406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:25.166090 14406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:25.166281 14406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:25.166851 14406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:25.167775 14406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:25.167843 14406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.167912 14406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:25.167963 14406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.177474 14406 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:41901
I20260925 22:50:25.177472 14533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:41901 every 8 connection(s)
I20260925 22:50:25.178445 14406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:50:25.184155 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14406
I20260925 22:50:25.184279 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:50:25.184643 4974 external_mini_cluster.cc:1428] Setting key 6a6dc3a67ab7d1bf2b51dfb6cb2e2681
I20260925 22:50:25.186697 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--builtin_ntp_servers=127.4.219.148:42255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:50:25.189319 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.180310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:39169 (local address 127.4.219.190:33403)
0925 22:50:25.180479 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:25.180482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:25.181633 (+ 1151us) server_negotiation.cc:407] Connection header received
0925 22:50:25.183277 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:25.183281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:25.183345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:25.183486 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:25.184584 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.185387 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.187005 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.187250 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.187343 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.188804 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:25.188807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:25.188810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:25.188823 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:25.188851 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:25.188860 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:25.188944 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:25.189118 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 22:50:25.189180 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:50:25.190213 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.180754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33403 (local address 127.4.219.130:39169)
0925 22:50:25.181460 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:25.181503 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:25.183006 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:25.183674 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:25.183682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:25.183772 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:25.184401 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:25.184420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.185528 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.185531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:25.186816 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:25.186831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.187409 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.187417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:25.187655 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.188466 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:25.188468 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:25.188507 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:25.188645 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:25.188946 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:25.188958 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:25.188966 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:50:25.189118 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 22:50:25.189415 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260925 22:50:25.191593 14534 heartbeater.cc:347] Connected to a master server at 127.4.219.190:33403
I20260925 22:50:25.191830 14534 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:25.192348 14534 heartbeater.cc:511] Master 127.4.219.190:33403 requested a full tablet report, sending...
I20260925 22:50:25.193558 14215 ts_manager.cc:194] Registered new tserver with Master: 5f735c69a61346378eb84f92cd1bf7de (127.4.219.130:41901)
I20260925 22:50:25.194545 14215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.130:39169
W20260925 22:50:25.306216 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:25.306488 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:25.306524 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:25.310622 14537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:25.310688 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:25.310741 14537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:25.315461 14537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:33403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:25.316672 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:25.318113 14537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:25.327868 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.327853 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.327960 14537 server_base.cc:1034] running on GCE node
W20260925 22:50:25.327903 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.328630 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:25.329300 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:25.330464 14537 hybrid_clock.cc:648] HybridClock initialized: now 1790376625330418 us; error 59 us; skew 500 ppm
I20260925 22:50:25.332345 14537 webserver.cc:460] Webserver started at http://127.4.219.131:38711/ using document root <none> and password file <none>
I20260925 22:50:25.332999 14537 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:25.333061 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:25.333272 14537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:25.335292 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "08175f9101814d61ae913222a7e1733a"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "3d62e27cff3f6569331454f17c835722"
server_key_iv: "e4c48022afdf711e1163f4c3f9314dda"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.335843 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "08175f9101814d61ae913222a7e1733a"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "3d62e27cff3f6569331454f17c835722"
server_key_iv: "e4c48022afdf711e1163f4c3f9314dda"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.339473 14537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:50:25.342147 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.343591 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:50:25.343688 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "08175f9101814d61ae913222a7e1733a"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "3d62e27cff3f6569331454f17c835722"
server_key_iv: "e4c48022afdf711e1163f4c3f9314dda"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.343782 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:25.359141 14537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:25.359800 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:25.359948 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:25.360481 14537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:25.361438 14537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:25.361483 14537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.361522 14537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:25.361537 14537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.370029 14537 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:37439
I20260925 22:50:25.370056 14664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:37439 every 8 connection(s)
I20260925 22:50:25.370957 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:50:25.373371 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14537
I20260925 22:50:25.373456 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:50:25.373739 4974 external_mini_cluster.cc:1428] Setting key 1748c856d5154f43193e7edb56a97d08
I20260925 22:50:25.380255 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.372827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:53237 (local address 127.4.219.190:33403)
0925 22:50:25.372960 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:25.372963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:25.374188 (+ 1225us) server_negotiation.cc:407] Connection header received
0925 22:50:25.375827 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:25.375830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:25.375872 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:25.375935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:25.376838 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.377353 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.378466 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.378669 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.378740 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.379843 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:25.379845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:25.379847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:25.379859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:25.379875 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:25.379882 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:25.379938 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:25.380094 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:50:25.380142 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:25.381167 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.373255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33403 (local address 127.4.219.131:53237)
0925 22:50:25.374020 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:25.374072 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:25.375636 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:25.376084 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:25.376090 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:25.376170 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:25.376710 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:25.376725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.377470 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.377473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:25.378362 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:25.378371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:25.378791 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:25.378794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:25.378950 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:25.379607 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:25.379609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:25.379648 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:25.379713 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:25.379959 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:25.379966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:25.379970 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:50:25.380072 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:50:25.380293 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
I20260925 22:50:25.382375 14665 heartbeater.cc:347] Connected to a master server at 127.4.219.190:33403
I20260925 22:50:25.382668 14665 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:25.383224 14665 heartbeater.cc:511] Master 127.4.219.190:33403 requested a full tablet report, sending...
I20260925 22:50:25.384302 14215 ts_manager.cc:194] Registered new tserver with Master: 08175f9101814d61ae913222a7e1733a (127.4.219.131:37439)
I20260925 22:50:25.384917 14215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.131:53237
I20260925 22:50:25.387473 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:50:25.390465 14254 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:25.389402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.4.219.190:33403)
0925 22:50:25.389593 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:25.389598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:25.389744 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:50:25.389912 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:25.389916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:25.389967 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:25.390052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:25.390293 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48562: BlockingRecv error: recv got EOF from 127.0.0.1:48562 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 22:50:25.390652 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14275
I20260925 22:50:25.396667 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14406
I20260925 22:50:25.401700 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14537
I20260925 22:50:25.406919 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14184
2026-09-25T22:50:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (850 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:25Z Disabled control of system clock
I20260925 22:50:25.435913 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37825
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:50:25.551170 14676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:25.551447 14676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:25.551486 14676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:25.555473 14676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:25.555536 14676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:25.555552 14676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:25.555568 14676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:25.555583 14676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:25.560189 14676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:37825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:25.561302 14676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:25.562700 14676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:25.571491 14682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.571511 14681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.571763 14676 server_base.cc:1034] running on GCE node
W20260925 22:50:25.571465 14684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.572438 14676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:25.573607 14676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:25.574815 14676 hybrid_clock.cc:648] HybridClock initialized: now 1790376625574790 us; error 34 us; skew 500 ppm
I20260925 22:50:25.577208 14676 webserver.cc:460] Webserver started at http://127.4.219.190:39127/ using document root <none> and password file <none>
I20260925 22:50:25.577937 14676 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:25.578017 14676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:25.578246 14676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:25.580324 14676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "9cb89643765b43859bf7d6377c76f151"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "e148a961376cb164b25873f75fbc7a17"
server_key_iv: "9de2d5f583dd5007663495c2f30d788b"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.580962 14676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "9cb89643765b43859bf7d6377c76f151"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "e148a961376cb164b25873f75fbc7a17"
server_key_iv: "9de2d5f583dd5007663495c2f30d788b"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.584697 14676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:25.587103 14690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.588438 14676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:25.588629 14676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "9cb89643765b43859bf7d6377c76f151"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "e148a961376cb164b25873f75fbc7a17"
server_key_iv: "9de2d5f583dd5007663495c2f30d788b"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.588753 14676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:25.593288 14676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:25.593894 14676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:25.594081 14676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:25.600703 14676 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:37825
I20260925 22:50:25.600739 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:37825 every 8 connection(s)
I20260925 22:50:25.601660 14676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:25.601899 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14676
I20260925 22:50:25.602012 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:50:25.602367 4974 external_mini_cluster.cc:1428] Setting key cb62834b1d469b4e987259dd7596503d
I20260925 22:50:25.605134 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:25.608749 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.603865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.4.219.190:37825)
0925 22:50:25.605114 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:25.605129 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:25.605165 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:50:25.605888 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:25.605905 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:25.606265 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:25.606634 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:25.607429 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:25.607433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:25.607445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:25.607455 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:25.607506 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:25.607522 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:25.607683 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:25.607751 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 22:50:25.607987 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":734,"threads_started":1}
I20260925 22:50:25.611171 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151: Bootstrap starting.
I20260925 22:50:25.613379 14743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:25.614159 14743 log.cc:824] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:25.616285 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151: No bootstrap required, opened a new log
I20260925 22:50:25.619820 14743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } }
I20260925 22:50:25.620059 14743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:25.620146 14743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb89643765b43859bf7d6377c76f151, State: Initialized, Role: FOLLOWER
I20260925 22:50:25.620592 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } }
I20260925 22:50:25.620736 14743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:25.620800 14743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:25.620899 14743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:25.621850 14743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } }
I20260925 22:50:25.622226 14743 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb89643765b43859bf7d6377c76f151; no voters:
I20260925 22:50:25.622602 14743 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:25.622727 14748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:25.622941 14748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [term 1 LEADER]: Becoming Leader. State: Replica: 9cb89643765b43859bf7d6377c76f151, State: Running, Role: LEADER
I20260925 22:50:25.623255 14748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } }
I20260925 22:50:25.623828 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:25.624910 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb89643765b43859bf7d6377c76f151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } } }
I20260925 22:50:25.625034 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:25.625339 14750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb89643765b43859bf7d6377c76f151. Latest consensus state: current_term: 1 leader_uuid: "9cb89643765b43859bf7d6377c76f151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb89643765b43859bf7d6377c76f151" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 37825 } } }
I20260925 22:50:25.625434 14750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151 [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:25.625514 14757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:25.628674 14757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:25.633899 14757 catalog_manager.cc:1347] Generated new cluster ID: 9da9be42c22f46f592954259fdbbdc3e
I20260925 22:50:25.633975 14757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:25.650085 14757 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:25.650426 14757 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:25.660604 14757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cb89643765b43859bf7d6377c76f151: Generated new TSK 0
I20260925 22:50:25.661336 14757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:50:25.670065 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--builtin_ntp_servers=127.4.219.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:50:25.790846 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:25.791124 14767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:25.791165 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:25.795174 14767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:25.795236 14767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:25.795292 14767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:25.799973 14767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:25.801139 14767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:25.802534 14767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:25.811817 14773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.811817 14775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.811844 14772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.812183 14767 server_base.cc:1034] running on GCE node
I20260925 22:50:25.812578 14767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:25.813172 14767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:25.814327 14767 hybrid_clock.cc:648] HybridClock initialized: now 1790376625814311 us; error 27 us; skew 500 ppm
I20260925 22:50:25.816116 14767 webserver.cc:460] Webserver started at http://127.4.219.129:46095/ using document root <none> and password file <none>
I20260925 22:50:25.816744 14767 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:25.816835 14767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:25.817070 14767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:25.819043 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "3e0600af922d432fa502824dc7dd3130"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "2c2b5b1599d59f999fa83dc396f1716e"
server_key_iv: "85450f142ac758427255e747d9316dff"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.819613 14767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "3e0600af922d432fa502824dc7dd3130"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "2c2b5b1599d59f999fa83dc396f1716e"
server_key_iv: "85450f142ac758427255e747d9316dff"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.823208 14767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 22:50:25.825778 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.827054 14767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:50:25.827190 14767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "3e0600af922d432fa502824dc7dd3130"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "2c2b5b1599d59f999fa83dc396f1716e"
server_key_iv: "85450f142ac758427255e747d9316dff"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.827303 14767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:25.831735 14767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:25.832315 14767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:25.832496 14767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:25.833024 14767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:25.833873 14767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:25.833943 14767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.834007 14767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:25.834056 14767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.842236 14767 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:42351
I20260925 22:50:25.842255 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:42351 every 8 connection(s)
I20260925 22:50:25.843106 14767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:25.846738 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14767
I20260925 22:50:25.846868 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:50:25.847177 4974 external_mini_cluster.cc:1428] Setting key 0601713fb3ffb5b3b58217e9bcdb5b44
I20260925 22:50:25.849290 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--builtin_ntp_servers=127.4.219.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:50:25.849722 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.845046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:47099 (local address 127.4.219.190:37825)
0925 22:50:25.845179 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:25.845198 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:25.846265 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 22:50:25.847784 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:25.847789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:25.847829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:25.847921 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:25.849165 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:25.849169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:25.849171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:25.849192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:25.849211 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:25.849219 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:25.849311 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:25.849554 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 22:50:25.849597 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:50:25.850530 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:25.845429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37825 (local address 127.4.219.129:47099)
0925 22:50:25.846119 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:25.846160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:25.847442 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:25.848115 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:25.848124 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:25.848236 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:25.848907 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:25.848909 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:25.848948 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:25.849016 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:25.849301 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:25.849309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:25.849315 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:50:25.849505 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 22:50:25.849757 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260925 22:50:25.851722 14895 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37825
I20260925 22:50:25.851948 14895 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:25.852355 14895 heartbeater.cc:511] Master 127.4.219.190:37825 requested a full tablet report, sending...
I20260925 22:50:25.854168 14707 ts_manager.cc:194] Registered new tserver with Master: 3e0600af922d432fa502824dc7dd3130 (127.4.219.129:42351)
W20260925 22:50:25.958420 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:25.958664 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:25.958702 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:25.962625 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:25.962674 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:25.962729 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:25.967250 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:25.968215 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:25.969619 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:25.978945 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.978960 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:25.978995 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:25.979332 14898 server_base.cc:1034] running on GCE node
I20260925 22:50:25.979830 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:25.980497 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:25.981715 14898 hybrid_clock.cc:648] HybridClock initialized: now 1790376625981677 us; error 46 us; skew 500 ppm
I20260925 22:50:25.983767 14898 webserver.cc:460] Webserver started at http://127.4.219.130:46297/ using document root <none> and password file <none>
I20260925 22:50:25.984404 14898 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:25.984503 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:25.984784 14898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:25.987000 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "1634656fd1004a0fb4a018b878eabafa"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "f8f2ad30dd5f7705b7c647522ba782d4"
server_key_iv: "9c938e8a99234145747a4b290b12408f"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.987622 14898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "1634656fd1004a0fb4a018b878eabafa"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "f8f2ad30dd5f7705b7c647522ba782d4"
server_key_iv: "9c938e8a99234145747a4b290b12408f"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.991614 14898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:50:25.994597 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:25.995895 14898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 22:50:25.996044 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "1634656fd1004a0fb4a018b878eabafa"
format_stamp: "Formatted at 2026-09-25 22:50:25 on dist-test-slave-kb1g"
server_key: "f8f2ad30dd5f7705b7c647522ba782d4"
server_key_iv: "9c938e8a99234145747a4b290b12408f"
server_key_version: "encryptionkey@0"
I20260925 22:50:25.996165 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:26.001120 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:26.001760 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:26.001966 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:26.002552 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:26.003500 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:26.003575 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:26.003651 14898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:26.003710 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:26.013329 14898 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:36995
I20260925 22:50:26.013347 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:36995 every 8 connection(s)
I20260925 22:50:26.014297 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:50:26.015585 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 14898
I20260925 22:50:26.015708 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:50:26.016013 4974 external_mini_cluster.cc:1428] Setting key d2d8871af7755d2f9dec6d78018da8fe
I20260925 22:50:26.018373 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--builtin_ntp_servers=127.4.219.148:37537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:50:26.021245 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:26.016332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:48401 (local address 127.4.219.190:37825)
0925 22:50:26.016498 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:26.016504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:26.017910 (+ 1406us) server_negotiation.cc:407] Connection header received
0925 22:50:26.019307 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:26.019310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:26.019349 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:26.019428 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:26.020783 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:26.020786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:26.020789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:26.020801 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:26.020824 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:26.020833 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:26.020930 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:26.021093 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 22:50:26.021133 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:50:26.022397 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:26.016962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37825 (local address 127.4.219.130:48401)
0925 22:50:26.017762 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:26.017814 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:26.019108 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:26.019609 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:26.019622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:26.019765 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:26.020418 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:26.020420 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:26.020479 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:26.020617 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:26.020933 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:26.020943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:26.020952 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:50:26.021126 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:50:26.021368 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260925 22:50:26.023684 15026 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37825
I20260925 22:50:26.023942 15026 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:26.024477 15026 heartbeater.cc:511] Master 127.4.219.190:37825 requested a full tablet report, sending...
I20260925 22:50:26.025739 14707 ts_manager.cc:194] Registered new tserver with Master: 1634656fd1004a0fb4a018b878eabafa (127.4.219.130:36995)
W20260925 22:50:26.132679 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:26.132951 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:26.132992 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:26.136955 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:26.137006 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:26.137063 15029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:26.141748 15029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:26.142791 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:26.144131 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:26.153824 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:26.153832 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:26.153903 15029 server_base.cc:1034] running on GCE node
W20260925 22:50:26.153841 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:26.154594 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:26.155236 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:26.156442 15029 hybrid_clock.cc:648] HybridClock initialized: now 1790376626156431 us; error 29 us; skew 500 ppm
I20260925 22:50:26.158432 15029 webserver.cc:460] Webserver started at http://127.4.219.131:38259/ using document root <none> and password file <none>
I20260925 22:50:26.159018 15029 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:26.159083 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:26.159253 15029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:26.161685 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "fdc941042bfd4dc88308a81070dcbe3a"
format_stamp: "Formatted at 2026-09-25 22:50:26 on dist-test-slave-kb1g"
server_key: "f7f520e931dddf5aec1fa0c32e0a8fcd"
server_key_iv: "5eafd4bc77bf7695eccc34e0ef58c3b5"
server_key_version: "encryptionkey@0"
I20260925 22:50:26.162317 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "fdc941042bfd4dc88308a81070dcbe3a"
format_stamp: "Formatted at 2026-09-25 22:50:26 on dist-test-slave-kb1g"
server_key: "f7f520e931dddf5aec1fa0c32e0a8fcd"
server_key_iv: "5eafd4bc77bf7695eccc34e0ef58c3b5"
server_key_version: "encryptionkey@0"
I20260925 22:50:26.166345 15029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:50:26.169296 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:26.170591 15029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:26.170715 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "fdc941042bfd4dc88308a81070dcbe3a"
format_stamp: "Formatted at 2026-09-25 22:50:26 on dist-test-slave-kb1g"
server_key: "f7f520e931dddf5aec1fa0c32e0a8fcd"
server_key_iv: "5eafd4bc77bf7695eccc34e0ef58c3b5"
server_key_version: "encryptionkey@0"
I20260925 22:50:26.170830 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:26.175482 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:26.176102 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:26.176278 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:26.176872 15029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:26.177763 15029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:26.177807 15029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:26.177873 15029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:26.177909 15029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:26.186643 15029 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:34491
I20260925 22:50:26.186663 15156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:34491 every 8 connection(s)
I20260925 22:50:26.187523 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:50:26.193909 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:26.189463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:51161 (local address 127.4.219.190:37825)
0925 22:50:26.189601 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:26.189605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:26.190671 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 22:50:26.192032 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:26.192036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:26.192109 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:26.192215 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:26.193410 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:26.193413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:26.193416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:50:26.193428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:26.193454 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:50:26.193463 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:26.193557 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:26.193730 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:50:26.193774 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:50:26.194785 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:26.189853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:37825 (local address 127.4.219.131:51161)
0925 22:50:26.190520 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:26.190569 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:26.191819 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:26.192379 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:26.192387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:26.192476 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:26.193162 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:50:26.193163 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:26.193204 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:50:26.193271 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:26.193576 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:26.193590 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:26.193595 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:50:26.193760 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:50:26.194001 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":101,"threads_started":1}
I20260925 22:50:26.195358 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 15029
I20260925 22:50:26.195470 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:50:26.195751 4974 external_mini_cluster.cc:1428] Setting key dddf0ac31bf7f570c6358ae90420a5e7
I20260925 22:50:26.196097 15157 heartbeater.cc:347] Connected to a master server at 127.4.219.190:37825
I20260925 22:50:26.196358 15157 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:26.196825 15157 heartbeater.cc:511] Master 127.4.219.190:37825 requested a full tablet report, sending...
I20260925 22:50:26.197811 14707 ts_manager.cc:194] Registered new tserver with Master: fdc941042bfd4dc88308a81070dcbe3a (127.4.219.131:34491)
I20260925 22:50:26.209570 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:50:26.212355 14746 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:50:26.211452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.4.219.190:37825)
0925 22:50:26.211627 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:26.211634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:26.211792 (+ 158us) server_negotiation.cc:407] Connection header received
0925 22:50:26.211855 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:26.211857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:26.211894 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:26.211982 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:26.212203 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52660: BlockingRecv error: recv got EOF from 127.0.0.1:52660 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:50:26.212419 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14767
I20260925 22:50:26.218020 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14898
I20260925 22:50:26.223562 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 15029
I20260925 22:50:26.228732 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 14676
2026-09-25T22:50:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:26 dist-test-slave-kb1g krb5kdc[15167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:26 dist-test-slave-kb1g krb5kdc[15167](info): set up 2 sockets
Sep 25 22:50:26 dist-test-slave-kb1g krb5kdc[15167](info): commencing operation
krb5kdc: starting...
W20260925 22:50:27.270154 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:50:27 dist-test-slave-kb1g krb5kdc[15167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:50:27.391546 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:40089
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:40089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:27.502076 15180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:27.502352 15180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:27.502394 15180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:27.506367 15180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:50:27.506431 15180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:27.506450 15180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:27.506467 15180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:50:27.506482 15180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:50:27.511958 15180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:40089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:27.513166 15180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:27.514178 15180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:27.523449 15186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:27.523450 15185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:27.523509 15188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:27.523866 15180 server_base.cc:1034] running on GCE node
I20260925 22:50:27.524458 15180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:27.525653 15180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:27.526846 15180 hybrid_clock.cc:648] HybridClock initialized: now 1790376627526818 us; error 48 us; skew 500 ppm
Sep 25 22:50:27 dist-test-slave-kb1g krb5kdc[15167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376627, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:27.536374 15180 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:50:27.537496 15180 webserver.cc:460] Webserver started at http://127.4.219.190:46155/ using document root <none> and password file <none>
I20260925 22:50:27.538087 15180 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:27.538177 15180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:27.538390 15180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:27.540438 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "519e8d1743bb4b39a36d4c799836440a"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "63fcdabd7334e70dae78f8985b322cfa"
server_key_iv: "8579ef1b1d9d3c628c788636c5b617d3"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.541056 15180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "519e8d1743bb4b39a36d4c799836440a"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "63fcdabd7334e70dae78f8985b322cfa"
server_key_iv: "8579ef1b1d9d3c628c788636c5b617d3"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.544755 15180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:50:27.547029 15195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:27.548363 15180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:27.548501 15180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "519e8d1743bb4b39a36d4c799836440a"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "63fcdabd7334e70dae78f8985b322cfa"
server_key_iv: "8579ef1b1d9d3c628c788636c5b617d3"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.548683 15180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:27.567131 15180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:27.570159 15180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:27.570361 15180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:27.577353 15180 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:40089
I20260925 22:50:27.577639 15247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:40089 every 8 connection(s)
I20260925 22:50:27.578344 15180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:50:27.581164 15248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:27.587637 15248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: Bootstrap starting.
I20260925 22:50:27.588032 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 15180
I20260925 22:50:27.588158 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:50:27.588428 4974 external_mini_cluster.cc:1428] Setting key 49d6f097591ecd278452d2b2711806d0
I20260925 22:50:27.590281 15248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:27.591032 15248 log.cc:824] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:27.593289 15248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: No bootstrap required, opened a new log
Sep 25 22:50:27 dist-test-slave-kb1g krb5kdc[15167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:27.597959 15248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } }
I20260925 22:50:27.598213 15248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:27.598281 15248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519e8d1743bb4b39a36d4c799836440a, State: Initialized, Role: FOLLOWER
I20260925 22:50:27.598733 15248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } }
I20260925 22:50:27.598904 15248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:50:27.598999 15248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:50:27.599116 15248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:27.600323 15248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } }
I20260925 22:50:27.600714 15248 leader_election.cc:302] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519e8d1743bb4b39a36d4c799836440a; no voters:
I20260925 22:50:27.601068 15248 leader_election.cc:288] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:50:27.601133 15253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:27.601320 15253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [term 1 LEADER]: Becoming Leader. State: Replica: 519e8d1743bb4b39a36d4c799836440a, State: Running, Role: LEADER
I20260925 22:50:27.601626 15253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } }
I20260925 22:50:27.602406 15248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:50:27.603914 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:27.589791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.4.219.190:40089)
0925 22:50:27.590208 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:27.590224 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:27.590258 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:50:27.590928 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:27.590945 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:27.591337 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:27.591738 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:27.592674 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.593733 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.594872 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.595322 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.595434 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:27.598303 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:27.598312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:27.598333 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:27.598386 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:27.600455 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:27.600935 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:27.600940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:27.600944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:27.601060 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:27.601534 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:27.601537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:27.601539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:27.602010 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:27.602149 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:27.602645 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 22:50:27.602964 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260925 22:50:27.605048 15254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519e8d1743bb4b39a36d4c799836440a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } } }
I20260925 22:50:27.605173 15254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [sys.catalog]: This master's current role is: LEADER
I20260925 22:50:27.605273 15255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519e8d1743bb4b39a36d4c799836440a. Latest consensus state: current_term: 1 leader_uuid: "519e8d1743bb4b39a36d4c799836440a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519e8d1743bb4b39a36d4c799836440a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 40089 } } }
I20260925 22:50:27.605360 15255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a [sys.catalog]: This master's current role is: LEADER
W20260925 22:50:27.607182 15267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:50:27.607256 15267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:50:27.607338 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:50:27.608103 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:50:27.613551 15268 catalog_manager.cc:1347] Generated new cluster ID: 5080dc75395442ad94648a0ff4788b38
I20260925 22:50:27.613627 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:50:27.631799 15268 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:50:27.632936 15268 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:50:27.638562 15268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: Generated new TSK 0
I20260925 22:50:27.639355 15268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:50:27.675194 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:27.786283 15274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:27.786542 15274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:27.786585 15274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:27.790627 15274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:27.790684 15274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:27.790784 15274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:50:27.796023 15274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:27.797109 15274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:27.798192 15274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:27.807948 15282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:27.808002 15279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:27.807986 15280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:27.808290 15274 server_base.cc:1034] running on GCE node
I20260925 22:50:27.808777 15274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:27.809397 15274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:27.810555 15274 hybrid_clock.cc:648] HybridClock initialized: now 1790376627810525 us; error 44 us; skew 500 ppm
Sep 25 22:50:27 dist-test-slave-kb1g krb5kdc[15167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376627, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:27.813244 15274 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:50:27.814041 15274 webserver.cc:460] Webserver started at http://127.4.219.129:43257/ using document root <none> and password file <none>
I20260925 22:50:27.814543 15274 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:27.814599 15274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:27.814754 15274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:27.816712 15274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "13353ca5306c4142bcf64482ff6dbf41"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "bdc581ddcc7bbe5fd043513aea3b63c0"
server_key_iv: "21df4349036bca50a1f66fc73c0ad3aa"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.817198 15274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "13353ca5306c4142bcf64482ff6dbf41"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "bdc581ddcc7bbe5fd043513aea3b63c0"
server_key_iv: "21df4349036bca50a1f66fc73c0ad3aa"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.820676 15274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260925 22:50:27.823060 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:27.824262 15274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:27.824394 15274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "13353ca5306c4142bcf64482ff6dbf41"
format_stamp: "Formatted at 2026-09-25 22:50:27 on dist-test-slave-kb1g"
server_key: "bdc581ddcc7bbe5fd043513aea3b63c0"
server_key_iv: "21df4349036bca50a1f66fc73c0ad3aa"
server_key_version: "encryptionkey@0"
I20260925 22:50:27.824507 15274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:27.843297 15274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:27.846935 15274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:27.847141 15274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:27.847733 15274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:27.848624 15274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:27.848706 15274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:27.848763 15274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:27.848795 15274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:27.858657 15274 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:34099
I20260925 22:50:27.858678 15402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:34099 every 8 connection(s)
I20260925 22:50:27.859582 15274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:50:27.861706 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 15274
I20260925 22:50:27.861837 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:50:27.862136 4974 external_mini_cluster.cc:1428] Setting key 97efabf7e6519475fa697b10c01149ea
Sep 25 22:50:27 dist-test-slave-kb1g krb5kdc[15167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376627, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:50:27.873968 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:27.861526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:41973 (local address 127.4.219.190:40089)
0925 22:50:27.861694 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:27.861700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:27.862858 (+ 1158us) server_negotiation.cc:407] Connection header received
0925 22:50:27.864036 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:27.864041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:27.864101 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:27.864192 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:27.865693 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.866212 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.867397 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.867637 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.867750 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:27.870469 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:27.870472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:27.870474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:27.870496 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:27.871969 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:27.872451 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:27.872456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:27.872458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:27.872586 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:27.872931 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:27.872934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:27.872935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:27.873114 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:27.873220 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:27.873686 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 22:50:27.873806 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:50:27.874817 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:27.861996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:40089 (local address 127.4.219.129:41973)
0925 22:50:27.862497 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:27.862556 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:27.863830 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:27.864387 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:27.864393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:27.864820 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:27.865537 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:27.865560 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.866377 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.866381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:27.867260 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:27.867268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:27.867784 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:27.867788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:27.867954 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:27.868635 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:27.868650 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:27.870271 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:27.872081 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:27.872087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:27.872101 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:27.872351 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:27.872685 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:27.872687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:27.872689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:27.872822 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:27.873216 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:27.873222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:27.873463 (+ 241us) client_negotiation.cc:769] Sending connection context
0925 22:50:27.873650 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:50:27.873977 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":142,"threads_started":1}
I20260925 22:50:27.876015 15403 heartbeater.cc:347] Connected to a master server at 127.4.219.190:40089
I20260925 22:50:27.876260 15403 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:27.876838 15403 heartbeater.cc:511] Master 127.4.219.190:40089 requested a full tablet report, sending...
I20260925 22:50:27.878486 15212 ts_manager.cc:194] Registered new tserver with Master: 13353ca5306c4142bcf64482ff6dbf41 (127.4.219.129:34099)
I20260925 22:50:27.880182 15212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:41973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:50:27.891985 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:28.002072 15408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:28.002414 15408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:28.002460 15408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:28.006817 15408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:28.006872 15408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:28.006932 15408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:50:28.012147 15408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:28.013226 15408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:28.014175 15408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:28.023661 15414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:28.023833 15408 server_base.cc:1034] running on GCE node
W20260925 22:50:28.023716 15416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:28.023830 15413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:28.024502 15408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:28.025179 15408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:28.026355 15408 hybrid_clock.cc:648] HybridClock initialized: now 1790376628026320 us; error 50 us; skew 500 ppm
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376628, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:28.029261 15408 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:50:28.030125 15408 webserver.cc:460] Webserver started at http://127.4.219.130:40689/ using document root <none> and password file <none>
I20260925 22:50:28.030675 15408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:28.030733 15408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:28.030938 15408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:28.032986 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "ce6c6be07e89447d9c847922c347d0c1"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "81b054c3c1818c091488be6f90b0bab3"
server_key_iv: "83fb0fc0b5845032a0da99101be1beb3"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.033542 15408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "ce6c6be07e89447d9c847922c347d0c1"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "81b054c3c1818c091488be6f90b0bab3"
server_key_iv: "83fb0fc0b5845032a0da99101be1beb3"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.037173 15408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:50:28.039486 15423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.040728 15408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:50:28.040818 15408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "ce6c6be07e89447d9c847922c347d0c1"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "81b054c3c1818c091488be6f90b0bab3"
server_key_iv: "83fb0fc0b5845032a0da99101be1beb3"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.040936 15408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:28.068882 15408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:28.072337 15408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:28.072599 15408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:28.073220 15408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:28.074036 15408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:28.074103 15408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.074173 15408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:28.074224 15408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.083616 15408 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44963
I20260925 22:50:28.083664 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44963 every 8 connection(s)
I20260925 22:50:28.084605 15408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:50:28.088738 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 15408
I20260925 22:50:28.088853 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:50:28.089172 4974 external_mini_cluster.cc:1428] Setting key ab9a7ee9ebaba6233ea29445ba9a9099
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376628, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:50:28.100201 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.086544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:37075 (local address 127.4.219.190:40089)
0925 22:50:28.086679 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.086682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.087596 (+ 914us) server_negotiation.cc:407] Connection header received
0925 22:50:28.089095 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.089098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.089155 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.089234 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:28.090755 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.091617 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.092916 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.093364 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.093533 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.095952 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:28.095959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:28.095965 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:28.096006 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:28.097955 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.098469 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.098474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.098476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.098577 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.098920 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.098924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.098926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.099202 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:28.099337 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.099821 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.099958 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:50:28.101020 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.086991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:40089 (local address 127.4.219.130:37075)
0925 22:50:28.087434 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.087478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.088846 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.089406 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.089413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.089887 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:28.090578 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.090600 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.091747 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.091749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.092789 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.092799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.093527 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.093529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.093667 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.094293 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:28.094310 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:28.095757 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:28.098081 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:28.098087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:28.098100 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:28.098360 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:28.098687 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:28.098689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:28.098691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:28.098821 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:28.099318 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:28.099324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:28.099568 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.099725 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.099985 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260925 22:50:28.102428 15537 heartbeater.cc:347] Connected to a master server at 127.4.219.190:40089
I20260925 22:50:28.102713 15537 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:28.103339 15537 heartbeater.cc:511] Master 127.4.219.190:40089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:50:28.104434 15212 ts_manager.cc:194] Registered new tserver with Master: ce6c6be07e89447d9c847922c347d0c1 (127.4.219.130:44963)
I20260925 22:50:28.105055 15212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:37075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:50:28.120041 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:50:28.231526 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:50:28.231796 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:50:28.231839 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:50:28.235906 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:50:28.235965 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:50:28.236023 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:50:28.241263 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:40089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:50:28.242337 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:50:28.243336 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:50:28.253374 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:50:28.253398 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:28.253537 15542 server_base.cc:1034] running on GCE node
W20260925 22:50:28.253734 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:50:28.254187 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:50:28.254917 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:50:28.256121 15542 hybrid_clock.cc:648] HybridClock initialized: now 1790376628256101 us; error 37 us; skew 500 ppm
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376628, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:50:28.259006 15542 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:50:28.260032 15542 webserver.cc:460] Webserver started at http://127.4.219.131:46321/ using document root <none> and password file <none>
I20260925 22:50:28.260704 15542 fs_manager.cc:359] Metadata directory not provided
I20260925 22:50:28.260804 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:50:28.261066 15542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:50:28.263314 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "31f644b381a34e96935fe320330f0e95"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "a8ed5332a93ba4d67e225818141e8992"
server_key_iv: "3564bccfb45b2aea67750aa64c6d7eeb"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.263931 15542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "31f644b381a34e96935fe320330f0e95"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "a8ed5332a93ba4d67e225818141e8992"
server_key_iv: "3564bccfb45b2aea67750aa64c6d7eeb"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.267977 15542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:50:28.270545 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.271835 15542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:50:28.272003 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "31f644b381a34e96935fe320330f0e95"
format_stamp: "Formatted at 2026-09-25 22:50:28 on dist-test-slave-kb1g"
server_key: "a8ed5332a93ba4d67e225818141e8992"
server_key_iv: "3564bccfb45b2aea67750aa64c6d7eeb"
server_key_version: "encryptionkey@0"
I20260925 22:50:28.272118 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:50:28.292240 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:50:28.295233 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:50:28.295446 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:50:28.296047 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:50:28.296942 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:50:28.297022 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.297094 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:50:28.297148 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:50:28.306448 15542 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:42815
I20260925 22:50:28.306463 15670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:42815 every 8 connection(s)
I20260925 22:50:28.307363 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376628, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:50:28.316789 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 15542
I20260925 22:50:28.316910 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:50:28.317255 4974 external_mini_cluster.cc:1428] Setting key 82c7791883118efc540872323e34a3b8
I20260925 22:50:28.320695 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.309205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:38135 (local address 127.4.219.190:40089)
0925 22:50:28.309344 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.309347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.310136 (+ 789us) server_negotiation.cc:407] Connection header received
0925 22:50:28.311487 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.311490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.311544 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.311619 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:28.312891 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.313415 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.314473 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.314679 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.314754 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.317109 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:28.317114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:28.317118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:28.317168 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:28.318745 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.319238 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.319243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.319245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.319322 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.319650 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.319655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.319657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.319829 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:28.319934 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.320401 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.320521 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:50:28.321480 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.309531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:40089 (local address 127.4.219.131:38135)
0925 22:50:28.309970 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.310012 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.311278 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.311786 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.311792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.312141 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:50:28.312755 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.312771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.313518 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.313520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.314350 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.314358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.314802 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.314804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.314936 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.315472 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:50:28.315487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:50:28.316928 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:50:28.318861 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:28.318867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:28.318881 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:28.319136 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:28.319413 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:50:28.319417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:50:28.319419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:50:28.319558 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:50:28.319930 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:50:28.319936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:50:28.320181 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.320363 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.320685 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260925 22:50:28.322631 15671 heartbeater.cc:347] Connected to a master server at 127.4.219.190:40089
I20260925 22:50:28.322863 15671 heartbeater.cc:464] Registering TS with master...
I20260925 22:50:28.323329 15671 heartbeater.cc:511] Master 127.4.219.190:40089 requested a full tablet report, sending...
I20260925 22:50:28.324327 15212 ts_manager.cc:194] Registered new tserver with Master: 31f644b381a34e96935fe320330f0e95 (127.4.219.131:42815)
I20260925 22:50:28.325012 15212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:38135
I20260925 22:50:28.331591 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:50:28.341054 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.333715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.4.219.190:40089)
0925 22:50:28.333850 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.333854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.334050 (+ 196us) server_negotiation.cc:407] Connection header received
0925 22:50:28.334149 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.334151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.334190 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.334259 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:50:28.334926 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.335469 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.336417 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.336709 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.336785 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.338158 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:50:28.338161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:50:28.338164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:50:28.338185 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:50:28.339480 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.339894 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.339897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.339898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.339969 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:50:28.340275 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:50:28.340278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:50:28.340279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:50:28.340474 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:50:28.340637 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.340836 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.340927 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:50:28.343885 15212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:50:28.347018 15212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:50:28.361502 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.355652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44963 (local address 127.0.0.1:59592)
0925 22:50:28.356017 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.356065 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.356202 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.357296 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.357303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.357338 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:28.357562 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.357571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.358566 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.358572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.359800 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.359810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.360944 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.360947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.361049 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.361085 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.361231 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.361360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":45,"spinlock_wait_cycles":3200,"thread_start_us":115,"threads_started":1}
I20260925 22:50:28.361510 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.356185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.4.219.130:44963)
0925 22:50:28.356507 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.356519 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.356563 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:50:28.356641 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.356647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.357089 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.357409 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:28.357683 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.358440 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.359943 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.360820 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.360944 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.361093 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.361261 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.361353 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":149,"threads_started":1}
I20260925 22:50:28.361783 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.356114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:42815 (local address 127.0.0.1:41470)
0925 22:50:28.356459 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.356472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.356579 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.357343 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.357346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.357362 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:28.357695 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.357700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.359089 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.359093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.360319 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.360338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.361383 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.361387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.361510 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.361531 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.361593 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.361665 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":256,"spinlock_wait_cycles":1408,"thread_start_us":84,"threads_started":1}
I20260925 22:50:28.362360 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.355414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34099 (local address 127.0.0.1:40240)
0925 22:50:28.356015 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.356065 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.356200 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.357310 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.357312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.357332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:28.357598 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.357605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.359116 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.359119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.360226 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.360235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.361383 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.361387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.362041 (+ 654us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.362070 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.362152 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.362225 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":568,"spinlock_wait_cycles":10240,"thread_start_us":60,"threads_started":1}
I20260925 22:50:28.362421 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.355586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.4.219.129:34099)
0925 22:50:28.356828 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.356843 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.356865 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:50:28.356937 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.356943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.357131 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.357333 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:28.357814 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.358951 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.360346 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.361239 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.361457 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.361600 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.362190 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.362269 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":95,"threads_started":1}
I20260925 22:50:28.363045 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.356522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.4.219.131:42815)
0925 22:50:28.356954 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.356964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.356981 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:50:28.357054 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.357058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.357204 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.357591 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:28.357778 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.358939 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.360454 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.361239 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.362613 (+ 1374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.362752 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.362801 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.362907 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":124,"threads_started":1}
I20260925 22:50:28.365254 15337 tablet_service.cc:1467] Processing CreateTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e (DEFAULT_TABLE table=test-table [id=978185a860454ca9b805690c533b6da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:28.365263 15471 tablet_service.cc:1467] Processing CreateTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e (DEFAULT_TABLE table=test-table [id=978185a860454ca9b805690c533b6da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:28.366240 15471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1324b174ec3848d4a2fe6896f9f2729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:28.366261 15337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1324b174ec3848d4a2fe6896f9f2729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:28.366525 15605 tablet_service.cc:1467] Processing CreateTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e (DEFAULT_TABLE table=test-table [id=978185a860454ca9b805690c533b6da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:50:28.367442 15605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1324b174ec3848d4a2fe6896f9f2729e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:50:28.372233 15688 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: Bootstrap starting.
I20260925 22:50:28.373874 15689 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Bootstrap starting.
I20260925 22:50:28.373874 15690 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Bootstrap starting.
I20260925 22:50:28.374703 15688 tablet_bootstrap.cc:654] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:28.376101 15688 log.cc:824] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:28.376225 15689 tablet_bootstrap.cc:654] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:28.376478 15690 tablet_bootstrap.cc:654] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Neither blocks nor log segments found. Creating new log.
I20260925 22:50:28.377103 15689 log.cc:824] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:28.377621 15690 log.cc:824] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Log is configured to *not* fsync() on all Append() calls
I20260925 22:50:28.378386 15688 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: No bootstrap required, opened a new log
I20260925 22:50:28.378583 15688 ts_tablet_manager.cc:1397] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:50:28.379189 15690 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: No bootstrap required, opened a new log
I20260925 22:50:28.379195 15689 tablet_bootstrap.cc:492] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: No bootstrap required, opened a new log
I20260925 22:50:28.379330 15690 ts_tablet_manager.cc:1397] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:50:28.379519 15689 ts_tablet_manager.cc:1397] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:50:28.382107 15690 raft_consensus.cc:348] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.382360 15690 raft_consensus.cc:374] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:28.382400 15690 raft_consensus.cc:729] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f644b381a34e96935fe320330f0e95, State: Initialized, Role: FOLLOWER
I20260925 22:50:28.382416 15688 raft_consensus.cc:348] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.382661 15688 raft_consensus.cc:374] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:28.382718 15688 raft_consensus.cc:729] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13353ca5306c4142bcf64482ff6dbf41, State: Initialized, Role: FOLLOWER
I20260925 22:50:28.382800 15690 consensus_queue.cc:260] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.383133 15688 consensus_queue.cc:260] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.383412 15671 heartbeater.cc:503] Master 127.4.219.190:40089 was elected leader, sending a full tablet report...
I20260925 22:50:28.383769 15690 ts_tablet_manager.cc:1428] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 22:50:28.383874 15689 raft_consensus.cc:348] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.384044 15688 ts_tablet_manager.cc:1428] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:50:28.384110 15689 raft_consensus.cc:374] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:50:28.384159 15689 raft_consensus.cc:729] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6c6be07e89447d9c847922c347d0c1, State: Initialized, Role: FOLLOWER
I20260925 22:50:28.384349 15403 heartbeater.cc:503] Master 127.4.219.190:40089 was elected leader, sending a full tablet report...
I20260925 22:50:28.384604 15689 consensus_queue.cc:260] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.385552 15537 heartbeater.cc:503] Master 127.4.219.190:40089 was elected leader, sending a full tablet report...
I20260925 22:50:28.386631 15689 ts_tablet_manager.cc:1428] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:50:28.510649 15695 raft_consensus.cc:482] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:50:28.510872 15695 raft_consensus.cc:504] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.512027 15695 leader_election.cc:288] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce6c6be07e89447d9c847922c347d0c1 (127.4.219.130:44963), 31f644b381a34e96935fe320330f0e95 (127.4.219.131:42815)
I20260925 22:50:28.516175 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.512556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:40049 (local address 127.4.219.130:44963)
0925 22:50:28.512683 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.512686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.512987 (+ 301us) server_negotiation.cc:407] Connection header received
0925 22:50:28.513173 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.513177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.513237 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.513291 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:28.513650 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.514157 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515187 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.515610 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515745 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.515806 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.515984 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.516048 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:50:28.516188 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.512465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44963 (local address 127.4.219.129:40049)
0925 22:50:28.512881 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.512899 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.513011 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.513417 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.513419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.513435 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:28.513536 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.513540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.514273 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.514278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.515059 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.515068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.515748 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.515861 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.515879 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.515969 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.516040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":232,"threads_started":1}
I20260925 22:50:28.516402 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.513173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:35381 (local address 127.4.219.131:42815)
0925 22:50:28.513295 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.513299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.513415 (+ 116us) server_negotiation.cc:407] Connection header received
0925 22:50:28.513585 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.513588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.513635 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.513717 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:50:28.513978 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.514453 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515469 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.515857 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.516030 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.516233 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.516296 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:50:28.516418 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.512986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:42815 (local address 127.4.219.129:35381)
0925 22:50:28.513304 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.513319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:50:28.513463 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:50:28.513728 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:50:28.513730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:50:28.513745 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:50:28.513868 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.513873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.514565 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.514571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.515345 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:50:28.515353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.515980 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.515983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:50:28.516106 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.516133 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:50:28.516218 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:50:28.516283 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":107,"threads_started":1}
I20260925 22:50:28.517236 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1324b174ec3848d4a2fe6896f9f2729e" candidate_uuid: "13353ca5306c4142bcf64482ff6dbf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f644b381a34e96935fe320330f0e95" is_pre_election: true
I20260925 22:50:28.517233 15491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1324b174ec3848d4a2fe6896f9f2729e" candidate_uuid: "13353ca5306c4142bcf64482ff6dbf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c6be07e89447d9c847922c347d0c1" is_pre_election: true
I20260925 22:50:28.517534 15625 raft_consensus.cc:2454] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13353ca5306c4142bcf64482ff6dbf41 in term 0.
I20260925 22:50:28.517539 15491 raft_consensus.cc:2454] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13353ca5306c4142bcf64482ff6dbf41 in term 0.
I20260925 22:50:28.517978 15290 leader_election.cc:302] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13353ca5306c4142bcf64482ff6dbf41, ce6c6be07e89447d9c847922c347d0c1; no voters:
I20260925 22:50:28.518237 15695 raft_consensus.cc:2781] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:50:28.518311 15695 raft_consensus.cc:482] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:50:28.518342 15695 raft_consensus.cc:3037] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:28.519366 15695 raft_consensus.cc:504] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.519701 15695 leader_election.cc:288] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [CANDIDATE]: Term 1 election: Requested vote from peers ce6c6be07e89447d9c847922c347d0c1 (127.4.219.130:44963), 31f644b381a34e96935fe320330f0e95 (127.4.219.131:42815)
I20260925 22:50:28.520157 15491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1324b174ec3848d4a2fe6896f9f2729e" candidate_uuid: "13353ca5306c4142bcf64482ff6dbf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6c6be07e89447d9c847922c347d0c1"
I20260925 22:50:28.520234 15625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1324b174ec3848d4a2fe6896f9f2729e" candidate_uuid: "13353ca5306c4142bcf64482ff6dbf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f644b381a34e96935fe320330f0e95"
I20260925 22:50:28.520298 15491 raft_consensus.cc:3037] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:28.520381 15625 raft_consensus.cc:3037] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:50:28.521399 15625 raft_consensus.cc:2454] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13353ca5306c4142bcf64482ff6dbf41 in term 1.
I20260925 22:50:28.521449 15491 raft_consensus.cc:2454] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13353ca5306c4142bcf64482ff6dbf41 in term 1.
I20260925 22:50:28.521688 15290 leader_election.cc:302] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13353ca5306c4142bcf64482ff6dbf41, 31f644b381a34e96935fe320330f0e95; no voters:
I20260925 22:50:28.521859 15695 raft_consensus.cc:2781] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:50:28.522073 15695 raft_consensus.cc:686] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 1 LEADER]: Becoming Leader. State: Replica: 13353ca5306c4142bcf64482ff6dbf41, State: Running, Role: LEADER
I20260925 22:50:28.522439 15695 consensus_queue.cc:237] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } }
I20260925 22:50:28.525929 15211 catalog_manager.cc:5495] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13353ca5306c4142bcf64482ff6dbf41 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "13353ca5306c4142bcf64482ff6dbf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13353ca5306c4142bcf64482ff6dbf41" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 } health_report { overall_health: UNKNOWN } } }
I20260925 22:50:28.550299 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.546828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.4.219.129:34099)
0925 22:50:28.546940 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:50:28.546943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:50:28.546957 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:50:28.547030 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:50:28.547032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:50:28.547086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:50:28.547184 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:50:28.547379 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.547914 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.548849 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:50:28.549110 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:50:28.549207 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:50:28.549449 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:50:28.549910 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:50:28.550024 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:50:28.550073 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:50:28.550127 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 22:50:28.556358 15491 raft_consensus.cc:1261] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 1 FOLLOWER]: Refusing update from remote peer 13353ca5306c4142bcf64482ff6dbf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:28.556392 15625 raft_consensus.cc:1261] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 1 FOLLOWER]: Refusing update from remote peer 13353ca5306c4142bcf64482ff6dbf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:50:28.557137 15695 consensus_queue.cc:1035] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f644b381a34e96935fe320330f0e95" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 42815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:50:28.557412 15699 consensus_queue.cc:1035] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6c6be07e89447d9c847922c347d0c1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:50:28.558290 15672 tablet.cc:2369] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:50:28.565148 15703 mvcc.cc:205] Tried to move back new op lower bound from 7333382670559903744 to 7333382670430797824. Current Snapshot: MvccSnapshot[applied={T|T < 7333382670559903744}]
I20260925 22:50:28.573472 15212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:28.573643 15212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:50:28.576087 15212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519e8d1743bb4b39a36d4c799836440a: Sending DeleteTablet for 3 replicas of tablet 1324b174ec3848d4a2fe6896f9f2729e
I20260925 22:50:28.576823 15337 tablet_service.cc:1514] Processing DeleteTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40240
I20260925 22:50:28.576977 15471 tablet_service.cc:1514] Processing DeleteTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:59592
I20260925 22:50:28.577468 15713 tablet_replica.cc:330] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: stopping tablet replica
I20260925 22:50:28.577750 15713 raft_consensus.cc:2229] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:28.578047 15713 raft_consensus.cc:2258] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:28.579299 15713 ts_tablet_manager.cc:1905] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:50:28.581485 15713 ts_tablet_manager.cc:1918] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:28.581595 15713 log.cc:1196] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/1324b174ec3848d4a2fe6896f9f2729e
I20260925 22:50:28.581885 15713 ts_tablet_manager.cc:1939] T 1324b174ec3848d4a2fe6896f9f2729e P ce6c6be07e89447d9c847922c347d0c1: Deleting consensus metadata
I20260925 22:50:28.582991 15196 catalog_manager.cc:4841] TS ce6c6be07e89447d9c847922c347d0c1 (127.4.219.130:44963): tablet 1324b174ec3848d4a2fe6896f9f2729e (table test-table [id=978185a860454ca9b805690c533b6da5]) successfully deleted
I20260925 22:50:28.584795 15712 tablet_replica.cc:330] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41: stopping tablet replica
I20260925 22:50:28.584873 15605 tablet_service.cc:1514] Processing DeleteTablet for tablet 1324b174ec3848d4a2fe6896f9f2729e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:50:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:41470
I20260925 22:50:28.585237 15712 raft_consensus.cc:2229] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:50:28.585364 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 15274
I20260925 22:50:28.585670 15712 raft_consensus.cc:2258] T 1324b174ec3848d4a2fe6896f9f2729e P 13353ca5306c4142bcf64482ff6dbf41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:28.588760 15714 tablet_replica.cc:330] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: stopping tablet replica
I20260925 22:50:28.589098 15714 raft_consensus.cc:2229] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:50:28.589401 15714 raft_consensus.cc:2258] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:50:28.590997 15714 ts_tablet_manager.cc:1905] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:50:28.593917 15199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:34099 (error 108)
I20260925 22:50:28.594146 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 15408
I20260925 22:50:28.594280 15714 ts_tablet_manager.cc:1918] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:50:28.594364 15714 log.cc:1196] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/1324b174ec3848d4a2fe6896f9f2729e
I20260925 22:50:28.594676 15714 ts_tablet_manager.cc:1939] T 1324b174ec3848d4a2fe6896f9f2729e P 31f644b381a34e96935fe320330f0e95: Deleting consensus metadata
I20260925 22:50:28.595247 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:50:28.594951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:34099 (local address 127.0.0.1:40260)
0925 22:50:28.595116 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:50:28.595173 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 22:50:28.595454 15199 catalog_manager.cc:4584] TS 13353ca5306c4142bcf64482ff6dbf41 (127.4.219.129:34099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1324b174ec3848d4a2fe6896f9f2729e: Network error: Client connection negotiation failed: client connection to 127.4.219.129:34099: connect: Connection refused (error 111)
I20260925 22:50:28.595957 15196 catalog_manager.cc:4841] TS 31f644b381a34e96935fe320330f0e95 (127.4.219.131:42815): tablet 1324b174ec3848d4a2fe6896f9f2729e (table test-table [id=978185a860454ca9b805690c533b6da5]) successfully deleted
I20260925 22:50:28.601203 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 15542
I20260925 22:50:28.606917 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 15180
2026-09-25T22:50:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2378 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15718](info): set up 2 sockets
Sep 25 22:50:28 dist-test-slave-kb1g krb5kdc[15718](info): commencing operation
krb5kdc: starting...
W20260925 22:50:29.645567 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:50:29 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:50:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangeradmin_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangeradmin_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangerlookup_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangerlookup_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
I20260925 22:50:29.817382 4974 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskre8WLI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/postgres -l logfile start
2026-09-25 22:50:32.358 UTC [15748] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:50:32.358 UTC [15748] LOG: listening on IPv4 address "127.4.219.148", port 59749
2026-09-25 22:50:32.361 UTC [15748] LOG: listening on Unix socket "/tmp/.s.PGSQL.59749"
2026-09-25 22:50:32.397 UTC [15751] LOG: database system was shut down at 2026-09-25 22:50:30 UTC
2026-09-25 22:50:32.402 UTC [15748] LOG: database system is ready to accept connections
I20260925 22:50:33.393770 4974 mini_postgres.cc:96] Postgres bound to 59749
2026-09-25 22:50:33.399 UTC [15759] FATAL: database "slave" does not exist
127.4.219.148:59749 - accepting connections
I20260925 22:50:33.400703 4974 mini_ranger.cc:161] Starting Ranger...
I20260925 22:50:33.415436 4974 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:50:33.918848 4974 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:50:33.918927 4974 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin
2026-09-25 22:50:34,211 [I] DB FLAVOR :POSTGRES
2026-09-25 22:50:34,211 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:50:34,211 [I] Checking connection..
2026-09-25 22:50:34,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:50:34,482 [I] Checking connection passed.
2026-09-25 22:50:34,483 [I] --------- Verifying version history table ---------
2026-09-25 22:50:34,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:50:34,755 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:50:34,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:50:35,002 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:50:35,002 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:50:35,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:50:35.265 UTC [15823] WARNING: there is no transaction in progress
2026-09-25 22:50:35,271 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:50:35,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:50:35,516 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:50:35,517 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:50:35,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:50:35,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:50:36,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kb1g.c.gcp-upstream.internal','N') ;"
2026-09-25 22:50:36,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:50:36,524 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:50:36,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:50:36,771 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:50:36,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:50:37,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:50:37,256 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:50:37,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:50:38.279 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.298 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.316 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.333 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.351 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.371 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.431 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.438 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.455 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.466 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.478 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.493 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.503 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.510 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.518 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:38.526 UTC [15931] WARNING: there is no transaction in progress
2026-09-25 22:50:39,275 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:50:39,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:50:39,530 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:50:39,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:50:39,771 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:50:39,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:50:40,049 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:50:40,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:50:40,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:50:40,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kb1g.c.gcp-upstream.internal';"
2026-09-25 22:50:40,795 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:50:40,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:50:41,041 [I] DB_PATCHES have already been applied
I20260925 22:50:41.047704 4974 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskre8WLI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:50:41.047780 4974 mini_ranger.cc:193] Using host: 127.4.219.148
I20260925 22:50:41.051659 4974 mini_ranger.cc:227] Ranger admin URL: http://127.4.219.148:54739
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:50:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:50:42.180919 4974 mini_ranger.cc:160] Time spent starting Ranger: real 8.780s user 0.000s sys 0.010s
Sep 25, 2026 10:51:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:51:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:51:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:51:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:51:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:51:03.441929 4974 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangerkms_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/rangerkms_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.219.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.219.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:51:03 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:51:03.531301 4974 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:51:03.541242 4974 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:51:04.440320 4974 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:51:04.440421 4974 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms
I20260925 22:51:04.440523 4974 mini_ranger_kms.cc:227] Using postgres at 127.4.219.148:59749
2026-09-25 22:51:04,633 [I] DB FLAVOR :POSTGRES
2026-09-25 22:51:04,634 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:51:04,634 [I] Checking connection
2026-09-25 22:51:04,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:51:04,906 [I] connection success
2026-09-25 22:51:04,906 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:51:04,906 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:51:04,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:51:05,153 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:51:05,154 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:51:05,154 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:51:05,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:59749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:51:05,410 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:51:05.892077 4974 mini_ranger_kms.cc:310] Created kms service
I20260925 22:51:06.065892 4974 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:51:06.111104 4974 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:51:06 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:06.490020 4974 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:51:06.490176 4974 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms:/tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:51:06.490203 4974 mini_ranger_kms.cc:240] Using host: 127.4.219.148
I20260925 22:51:06.494020 4974 mini_ranger_kms.cc:276] Ranger KMS PID: 16111
I20260925 22:51:06.494170 4974 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.219.148:59861
Sep 25, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:51:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:51:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:51:08.832711 4974 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.301s user 0.011s sys 0.163s
I20260925 22:51:08.832841 4974 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:51:08.832947 4974 mini_ranger_kms.cc:401] 127.4.219.148:59861/kms/v1/keys
Sep 25 22:51:08 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:51:09 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
Sep 25 22:51:09 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
Sep 25 22:51:11 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:51:11.944792 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44189
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44189
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:59861/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:12.064775 16162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:12.065622 16162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:12.065802 16162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:12.073619 16162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:51:12.073719 16162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:12.073756 16162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:12.073796 16162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:51:12.073833 16162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:51:12.083798 16162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.219.148:59861/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44189
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:12.085471 16162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:12.086846 16162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:12.098384 16167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:12.098439 16162 server_base.cc:1034] running on GCE node
W20260925 22:51:12.098429 16168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:12.098758 16170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:12.099705 16162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:12.101456 16162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:12.102646 16162 hybrid_clock.cc:648] HybridClock initialized: now 1790376672102621 us; error 52 us; skew 500 ppm
Sep 25 22:51:12 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376672, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:12.105485 16162 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:51:12.106402 16162 webserver.cc:460] Webserver started at http://127.4.219.190:39183/ using document root <none> and password file <none>
I20260925 22:51:12.106945 16162 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:12.106998 16162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:12.107221 16162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:12 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376672, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:12.155205 16162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "93bf881e349641f897a0dade83d82311"
format_stamp: "Formatted at 2026-09-25 22:51:12 on dist-test-slave-kb1g"
server_key: "bd0be5f7782790272e09a7bbaebe766f"
server_key_iv: "4f8f53b2ea8785a7f8a0118d48ea67ec"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:12.155887 16162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "93bf881e349641f897a0dade83d82311"
format_stamp: "Formatted at 2026-09-25 22:51:12 on dist-test-slave-kb1g"
server_key: "bd0be5f7782790272e09a7bbaebe766f"
server_key_iv: "4f8f53b2ea8785a7f8a0118d48ea67ec"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:12.159875 16162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 22:51:12.176601 16177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:12.178157 16162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:51:12.178303 16162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "93bf881e349641f897a0dade83d82311"
format_stamp: "Formatted at 2026-09-25 22:51:12 on dist-test-slave-kb1g"
server_key: "bd0be5f7782790272e09a7bbaebe766f"
server_key_iv: "4f8f53b2ea8785a7f8a0118d48ea67ec"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:12.178434 16162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:12.196692 16162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:12.199302 16162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:12.199468 16162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:12.206779 16162 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44189
I20260925 22:51:12.206805 16229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44189 every 8 connection(s)
I20260925 22:51:12.207736 16162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:51:12.210497 16230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:51:12.212947 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 16162
I20260925 22:51:12.213081 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:51:12.213380 4974 external_mini_cluster.cc:1428] Setting key 9721cfdd520dba0d04238d9184945c45
I20260925 22:51:12.216450 16230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: Bootstrap starting.
I20260925 22:51:12.218557 16230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: Neither blocks nor log segments found. Creating new log.
I20260925 22:51:12.219261 16230 log.cc:824] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: Log is configured to *not* fsync() on all Append() calls
I20260925 22:51:12.221515 16230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: No bootstrap required, opened a new log
Sep 25 22:51:12 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:12.225088 16230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } }
I20260925 22:51:12.225387 16230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:51:12.225478 16230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93bf881e349641f897a0dade83d82311, State: Initialized, Role: FOLLOWER
I20260925 22:51:12.225950 16230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } }
I20260925 22:51:12.226117 16230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:51:12.226186 16230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:51:12.226280 16230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:51:12.227476 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:12.214814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.4.219.190:44189)
0925 22:51:12.215245 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:12.215259 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:12.215293 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:51:12.215909 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:12.215926 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:12.216261 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:12.216617 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:12.217397 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:12.218342 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:12.219225 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:12.219844 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:12.220011 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:12.222820 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:12.222825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:12.222838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:12.222864 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:12.224493 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:12.225000 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:12.225004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:12.225009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:12.225124 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:12.225422 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:12.225426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:12.225428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:12.225958 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:12.226160 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:12.226456 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 22:51:12.226739 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260925 22:51:12.227674 16230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } }
I20260925 22:51:12.227990 16230 leader_election.cc:302] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93bf881e349641f897a0dade83d82311; no voters:
I20260925 22:51:12.228240 16230 leader_election.cc:288] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:51:12.228340 16235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:51:12.229282 16230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:51:12.229508 16235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [term 1 LEADER]: Becoming Leader. State: Replica: 93bf881e349641f897a0dade83d82311, State: Running, Role: LEADER
I20260925 22:51:12.229811 16235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } }
I20260925 22:51:12.231473 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93bf881e349641f897a0dade83d82311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } } }
I20260925 22:51:12.231566 16230 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:51:12.231598 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [sys.catalog]: This master's current role is: LEADER
I20260925 22:51:12.231792 16236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93bf881e349641f897a0dade83d82311. Latest consensus state: current_term: 1 leader_uuid: "93bf881e349641f897a0dade83d82311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bf881e349641f897a0dade83d82311" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44189 } } }
I20260925 22:51:12.231874 16236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311 [sys.catalog]: This master's current role is: LEADER
I20260925 22:51:12.232028 16249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:51:12.237990 16249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:51:12.245461 16249 catalog_manager.cc:1347] Generated new cluster ID: 249c17a859e340fe9f3bbcf68784e9ad
I20260925 22:51:12.245533 16249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:51:12.250211 16249 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:51:12.251097 16249 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:51:12.259251 16249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: Generated new TSK 0
I20260925 22:51:12.259912 16249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:51:12.987825 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:59861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:13.129081 16275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:13.129328 16275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:13.129405 16275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:13.134258 16275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:13.134334 16275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:13.134423 16275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:51:13.142225 16275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.219.148:59861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:13.143497 16275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:13.144636 16275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:13.156203 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:13.156200 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:13.156690 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:13.157248 16275 server_base.cc:1034] running on GCE node
I20260925 22:51:13.157712 16275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:13.158476 16275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:13.159672 16275 hybrid_clock.cc:648] HybridClock initialized: now 1790376673159642 us; error 43 us; skew 500 ppm
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:13.163301 16275 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:51:13.164433 16275 webserver.cc:460] Webserver started at http://127.4.219.129:43355/ using document root <none> and password file <none>
I20260925 22:51:13.165033 16275 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:13.165094 16275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:13.165338 16275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:13.187747 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "f4830106851449c6a0ebb5638c48e15a"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "eccbd0971a2ec44a943135c1ff4c8b9e"
server_key_iv: "525c2b848b0c53ef81271c1d35b99d5b"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.188570 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "f4830106851449c6a0ebb5638c48e15a"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "eccbd0971a2ec44a943135c1ff4c8b9e"
server_key_iv: "525c2b848b0c53ef81271c1d35b99d5b"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.202152 16275 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.002s
I20260925 22:51:13.228137 16294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.229980 16275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:51:13.230144 16275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "f4830106851449c6a0ebb5638c48e15a"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "eccbd0971a2ec44a943135c1ff4c8b9e"
server_key_iv: "525c2b848b0c53ef81271c1d35b99d5b"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.230283 16275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:13.256464 16275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:13.269881 16275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:13.270171 16275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:13.270903 16275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:13.271996 16275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:13.272117 16275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.272243 16275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:13.272323 16275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.283056 16275 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:42611
I20260925 22:51:13.293967 16275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:51:13.298280 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 16275
I20260925 22:51:13.298431 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:51:13.298785 4974 external_mini_cluster.cc:1428] Setting key c6e1fabd3004ee60be1b1febd566a1b4
I20260925 22:51:13.295502 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:42611 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:13.346653 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:13.316691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:40593 (local address 127.4.219.190:44189)
0925 22:51:13.326820 (+ 10129us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:13.326826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:13.326842 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:51:13.326890 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:13.326893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:13.326949 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:13.327087 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:13.328935 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.329909 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.331996 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.332356 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.332446 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:13.338127 (+ 5681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:13.338132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:13.338135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:13.338173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:13.340173 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:13.340923 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:13.340927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:13.340929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:13.341027 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:13.341511 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:13.341515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:13.341516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:13.341765 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:13.341854 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:13.346214 (+ 4360us) server_negotiation.cc:299] Negotiation successful
0925 22:51:13.346378 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10027,"thread_start_us":83,"threads_started":1}
I20260925 22:51:13.349076 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:13.315453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44189 (local address 127.4.219.129:40593)
0925 22:51:13.324956 (+ 9503us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:13.325045 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:13.326219 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:13.327470 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:13.327481 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:13.327865 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:13.328750 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:13.328774 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.330077 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.330081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:13.331845 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:13.331857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.332692 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.332696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:13.332903 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:13.333656 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:13.333678 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:13.336316 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:13.340337 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:13.340346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:13.340366 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:13.340783 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:13.341140 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:13.341143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:13.341145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:13.341371 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:13.341928 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:13.341935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:13.342299 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 22:51:13.347255 (+ 4956us) client_negotiation.cc:240] Negotiation successful
0925 22:51:13.347730 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9176,"thread_start_us":161,"threads_started":1}
I20260925 22:51:13.354291 16408 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44189
I20260925 22:51:13.354660 16408 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:13.355394 16408 heartbeater.cc:511] Master 127.4.219.190:44189 requested a full tablet report, sending...
I20260925 22:51:13.357669 16194 ts_manager.cc:194] Registered new tserver with Master: f4830106851449c6a0ebb5638c48e15a (127.4.219.129:42611)
I20260925 22:51:13.359537 16194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:40593
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:51:13.374154 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:59861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:13.538543 16414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:13.539000 16414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:13.539120 16414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:13.545111 16414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:13.545194 16414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:13.545296 16414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:51:13.550977 16414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.219.148:59861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:13.552614 16414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:13.553727 16414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:13.566496 16423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:13.566496 16421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:13.567904 16414 server_base.cc:1034] running on GCE node
W20260925 22:51:13.568666 16420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:13.569326 16414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:13.570159 16414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:13.571501 16414 hybrid_clock.cc:648] HybridClock initialized: now 1790376673571461 us; error 49 us; skew 500 ppm
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:13.574683 16414 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:51:13.575760 16414 webserver.cc:460] Webserver started at http://127.4.219.130:44445/ using document root <none> and password file <none>
I20260925 22:51:13.576473 16414 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:13.576598 16414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:13.576859 16414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:13.597623 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "6016d1fad6ad4ecfa604afe4b99a01a1"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "49615764c9d62ff55aa619515fc209fa"
server_key_iv: "817cf958a9ab43b6c470e84751770574"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.598529 16414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "6016d1fad6ad4ecfa604afe4b99a01a1"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "49615764c9d62ff55aa619515fc209fa"
server_key_iv: "817cf958a9ab43b6c470e84751770574"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.603129 16414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:51:13.621855 16430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.623636 16414 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260925 22:51:13.623800 16414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "6016d1fad6ad4ecfa604afe4b99a01a1"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "49615764c9d62ff55aa619515fc209fa"
server_key_iv: "817cf958a9ab43b6c470e84751770574"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.623972 16414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:13.657944 16414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:13.661767 16414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:13.662213 16414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:13.662954 16414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:13.664124 16414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:13.664191 16414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.664266 16414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:13.664299 16414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:13.674971 16414 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:41455
I20260925 22:51:13.676170 16414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:51:13.678512 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 16414
I20260925 22:51:13.678654 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:51:13.678975 4974 external_mini_cluster.cc:1428] Setting key 634b7d4ee3fc05df708c337b75e823d0
I20260925 22:51:13.693394 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:41455 every 8 connection(s)
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:51:13.714202 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:13.695486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:41861 (local address 127.4.219.190:44189)
0925 22:51:13.695623 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:13.695628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:13.697547 (+ 1919us) server_negotiation.cc:407] Connection header received
0925 22:51:13.698813 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:13.698817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:13.698883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:13.698940 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:13.700673 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.701556 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.703306 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.703663 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.703748 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:13.708437 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:13.708443 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:13.708448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:13.708490 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:13.710412 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:13.712383 (+ 1971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:13.712386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:13.712387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:13.712462 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:13.712920 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:13.712924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:13.712926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:13.713120 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:13.713228 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:13.713839 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 22:51:13.713975 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:51:13.715278 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:13.696108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44189 (local address 127.4.219.130:41861)
0925 22:51:13.697385 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:13.697443 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:13.698564 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:13.699366 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:13.699377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:13.699736 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:13.700455 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:13.700477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.701711 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.701714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:13.703160 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:13.703172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:13.703959 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:13.703963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:13.704172 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:13.705765 (+ 1593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:13.705791 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:13.708211 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:13.711940 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:13.711947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:13.711966 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:13.712241 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:13.712591 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:13.712594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:13.712596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:13.712794 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:13.713238 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:13.713247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:13.713560 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 22:51:13.713787 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 22:51:13.714143 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":172,"threads_started":1}
I20260925 22:51:13.716809 16544 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44189
I20260925 22:51:13.717144 16544 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:13.717805 16544 heartbeater.cc:511] Master 127.4.219.190:44189 requested a full tablet report, sending...
I20260925 22:51:13.719080 16194 ts_manager.cc:194] Registered new tserver with Master: 6016d1fad6ad4ecfa604afe4b99a01a1 (127.4.219.130:41455)
I20260925 22:51:13.719853 16194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:41861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:51:13.735713 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:59861/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:13.907917 16549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:13.908279 16549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:13.908344 16549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:13.914839 16549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:13.914918 16549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:13.915020 16549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:51:13.922904 16549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.219.148:59861/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:13.924232 16549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:13.925598 16549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
W20260925 22:51:13.939074 16555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:13.939388 16557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:13.939810 16549 server_base.cc:1034] running on GCE node
W20260925 22:51:13.941493 16554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:13.941962 16549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:13.942756 16549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:13.952648 16549 hybrid_clock.cc:648] HybridClock initialized: now 1790376673952564 us; error 88 us; skew 500 ppm
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:13.960371 16549 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:51:13.961553 16549 webserver.cc:460] Webserver started at http://127.4.219.131:37807/ using document root <none> and password file <none>
I20260925 22:51:13.962261 16549 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:13.962342 16549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:13.962574 16549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:13 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:13.977886 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "ed346998e600401688328f7b555ee0d9"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "3a66ed00c0c863eecb8b925e586f4db7"
server_key_iv: "beaa16080fa7fec69c8763fba5625ebb"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.984244 16549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "ed346998e600401688328f7b555ee0d9"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "3a66ed00c0c863eecb8b925e586f4db7"
server_key_iv: "beaa16080fa7fec69c8763fba5625ebb"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:13.989945 16549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 22:51:14.020157 16564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:14.022147 16549 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260925 22:51:14.022293 16549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "ed346998e600401688328f7b555ee0d9"
format_stamp: "Formatted at 2026-09-25 22:51:13 on dist-test-slave-kb1g"
server_key: "3a66ed00c0c863eecb8b925e586f4db7"
server_key_iv: "beaa16080fa7fec69c8763fba5625ebb"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:14.022430 16549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:14.050674 16549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:14.054381 16549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:14.054638 16549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:14.055394 16549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:14.056476 16549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:14.056614 16549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:14.056710 16549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:14.056779 16549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:14.069497 16549 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:40411
I20260925 22:51:14.070715 16549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:51:14.075544 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 16549
I20260925 22:51:14.075665 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:51:14.075948 4974 external_mini_cluster.cc:1428] Setting key 104cc72aeae249c4e1a1b8747245679d
I20260925 22:51:14.072094 16677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:40411 every 8 connection(s)
Sep 25 22:51:14 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:14.095000 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.073853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:60435 (local address 127.4.219.190:44189)
0925 22:51:14.074003 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.074008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.081846 (+ 7838us) server_negotiation.cc:407] Connection header received
0925 22:51:14.083118 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.083121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.083175 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.083222 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:14.084905 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.085735 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.087437 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.087746 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.087869 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.090793 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:14.090797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:14.090799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:14.090828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:14.092659 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:14.093261 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:14.093264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:14.093266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:14.093349 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:14.093743 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:14.093746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:14.093748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:14.093962 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:14.094042 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.094707 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.094835 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:51:14.096864 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.074349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44189 (local address 127.4.219.131:60435)
0925 22:51:14.081661 (+ 7312us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.081740 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.082858 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.083663 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.083673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.084040 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:14.084746 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.084765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.085883 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.085886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.087279 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.087289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.087921 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.087925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.088120 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.088836 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:14.088856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:14.090593 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:14.092786 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:14.092792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:14.092810 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:14.093147 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:14.093446 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:14.093449 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:14.093451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:14.093637 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:14.094103 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:14.094110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:14.094421 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.095395 (+ 974us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.095759 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6586,"thread_start_us":131,"threads_started":1}
I20260925 22:51:14.098101 16678 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44189
I20260925 22:51:14.098345 16678 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:14.098917 16678 heartbeater.cc:511] Master 127.4.219.190:44189 requested a full tablet report, sending...
I20260925 22:51:14.100023 16194 ts_manager.cc:194] Registered new tserver with Master: ed346998e600401688328f7b555ee0d9 (127.4.219.131:40411)
I20260925 22:51:14.100780 16194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:60435
I20260925 22:51:14.106032 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:51:14.120853 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.112797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.4.219.190:44189)
0925 22:51:14.112939 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.112944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.113356 (+ 412us) server_negotiation.cc:407] Connection header received
0925 22:51:14.113522 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.113526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.113572 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.113674 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:14.114377 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.114946 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.115791 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.116028 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.116139 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.117630 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:14.117634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:14.117636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:14.117661 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:14.119161 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:14.119587 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:14.119592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:14.119594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:14.119679 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:14.119958 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:14.119963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:14.119965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:14.120144 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:14.120280 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.120524 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.120678 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:51:14.124362 16194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:51:14.127912 16194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:51:14.143870 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.138124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.4.219.129:42611)
0925 22:51:14.139607 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.139613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.139634 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:51:14.139695 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.139698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.139861 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.140035 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:51:14.140312 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.141128 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.142587 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.143154 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.143279 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.143437 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.143619 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.143717 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":70,"threads_started":1}
I20260925 22:51:14.143869 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.137320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:42611 (local address 127.0.0.1:50172)
0925 22:51:14.139140 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.139189 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.139314 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.140019 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.140025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.140048 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:51:14.140186 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.140201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.141245 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.141250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.142464 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.142474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.143288 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.143291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.143414 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.143456 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.143590 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.143713 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1641,"thread_start_us":116,"threads_started":1}
I20260925 22:51:14.147780 16342 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad5805959d4073aa5bda5fa2823733 (DEFAULT_TABLE table=test-table [id=718d5a442cef4c10a9fbc65fb146ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:51:14.149013 16342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad5805959d4073aa5bda5fa2823733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:51:14.149336 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.138249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.4.219.130:41455)
0925 22:51:14.138536 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.138552 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.144162 (+ 5610us) server_negotiation.cc:407] Connection header received
0925 22:51:14.144400 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.144406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.144678 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.144821 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:51:14.145206 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.146284 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.147729 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.148481 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.148601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.148753 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.149095 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.149194 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":115,"threads_started":1}
I20260925 22:51:14.149696 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.137712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:41455 (local address 127.0.0.1:37534)
0925 22:51:14.144061 (+ 6349us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.144077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.144237 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.144997 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.145000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.145016 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:51:14.145084 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.145089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.146436 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.146439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.147573 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.147584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.148847 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.148852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.148938 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.148960 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.149486 (+ 526us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.149557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6276,"mutex_wait_us":52,"thread_start_us":79,"threads_started":1}
I20260925 22:51:14.152700 16477 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad5805959d4073aa5bda5fa2823733 (DEFAULT_TABLE table=test-table [id=718d5a442cef4c10a9fbc65fb146ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:51:14.153713 16477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad5805959d4073aa5bda5fa2823733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:51:14.154224 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.138766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:40411 (local address 127.0.0.1:46202)
0925 22:51:14.144676 (+ 5910us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.144690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.144784 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.145675 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.145678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.145695 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:51:14.145776 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.145782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.147228 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.147234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.148466 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.148478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.153813 (+ 5335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.153817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.153903 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.153925 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.154017 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.154082 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5846,"mutex_wait_us":5,"thread_start_us":112,"threads_started":1}
I20260925 22:51:14.154623 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.139155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.4.219.131:40411)
0925 22:51:14.145069 (+ 5914us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.145082 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.145107 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:51:14.145184 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.145190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.145381 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.145532 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:51:14.145919 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.147089 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.152690 (+ 5601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.153514 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.153621 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.153753 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.154380 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.154482 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5783,"thread_start_us":117,"threads_started":1}
I20260925 22:51:14.157389 16612 tablet_service.cc:1467] Processing CreateTablet for tablet 7bad5805959d4073aa5bda5fa2823733 (DEFAULT_TABLE table=test-table [id=718d5a442cef4c10a9fbc65fb146ea17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:51:14.158371 16612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bad5805959d4073aa5bda5fa2823733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:51:14.168687 16698 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Bootstrap starting.
I20260925 22:51:14.169107 16697 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Bootstrap starting.
I20260925 22:51:14.169322 16696 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Bootstrap starting.
I20260925 22:51:14.171627 16698 tablet_bootstrap.cc:654] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Neither blocks nor log segments found. Creating new log.
I20260925 22:51:14.172571 16698 log.cc:824] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:51:14.173341 16697 tablet_bootstrap.cc:654] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Neither blocks nor log segments found. Creating new log.
I20260925 22:51:14.174136 16697 log.cc:824] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:51:14.175200 16696 tablet_bootstrap.cc:654] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Neither blocks nor log segments found. Creating new log.
I20260925 22:51:14.175951 16696 log.cc:824] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:51:14.176339 16698 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: No bootstrap required, opened a new log
I20260925 22:51:14.176586 16698 ts_tablet_manager.cc:1397] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:51:14.178068 16697 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: No bootstrap required, opened a new log
I20260925 22:51:14.178331 16697 ts_tablet_manager.cc:1397] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 22:51:14.180737 16698 raft_consensus.cc:348] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.181025 16698 raft_consensus.cc:374] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:51:14.181077 16698 raft_consensus.cc:729] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed346998e600401688328f7b555ee0d9, State: Initialized, Role: FOLLOWER
I20260925 22:51:14.181646 16698 consensus_queue.cc:260] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.182838 16698 ts_tablet_manager.cc:1428] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:51:14.183292 16678 heartbeater.cc:503] Master 127.4.219.190:44189 was elected leader, sending a full tablet report...
I20260925 22:51:14.184007 16697 raft_consensus.cc:348] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.184264 16697 raft_consensus.cc:374] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:51:14.184342 16697 raft_consensus.cc:729] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4830106851449c6a0ebb5638c48e15a, State: Initialized, Role: FOLLOWER
I20260925 22:51:14.186883 16696 tablet_bootstrap.cc:492] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: No bootstrap required, opened a new log
I20260925 22:51:14.187142 16696 ts_tablet_manager.cc:1397] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260925 22:51:14.188580 16697 consensus_queue.cc:260] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.189693 16697 ts_tablet_manager.cc:1428] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260925 22:51:14.190950 16408 heartbeater.cc:503] Master 127.4.219.190:44189 was elected leader, sending a full tablet report...
I20260925 22:51:14.191169 16696 raft_consensus.cc:348] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.191424 16696 raft_consensus.cc:374] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:51:14.191480 16696 raft_consensus.cc:729] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6016d1fad6ad4ecfa604afe4b99a01a1, State: Initialized, Role: FOLLOWER
I20260925 22:51:14.191893 16696 consensus_queue.cc:260] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.192976 16696 ts_tablet_manager.cc:1428] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:51:14.193336 16544 heartbeater.cc:503] Master 127.4.219.190:44189 was elected leader, sending a full tablet report...
W20260925 22:51:14.196408 16546 tablet.cc:2369] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:51:14 dist-test-slave-kb1g krb5kdc[15718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376673, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
W20260925 22:51:14.318043 16409 tablet.cc:2369] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:51:14.327296 16679 tablet.cc:2369] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:51:14.486327 16703 raft_consensus.cc:482] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:51:14.486593 16703 raft_consensus.cc:504] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.487876 16703 leader_election.cc:288] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed346998e600401688328f7b555ee0d9 (127.4.219.131:40411), 6016d1fad6ad4ecfa604afe4b99a01a1 (127.4.219.130:41455)
I20260925 22:51:14.493013 16695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.488258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:39505 (local address 127.4.219.131:40411)
0925 22:51:14.488390 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.488396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.488579 (+ 183us) server_negotiation.cc:407] Connection header received
0925 22:51:14.488818 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.488821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.488887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.488943 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:51:14.489390 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.490123 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.491703 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.492287 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.492384 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.492437 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.492782 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.492859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:51:14.493425 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.488733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:58749 (local address 127.4.219.130:41455)
0925 22:51:14.488863 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.488878 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.489405 (+ 527us) server_negotiation.cc:407] Connection header received
0925 22:51:14.489575 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.489579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.489626 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.489673 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:51:14.490036 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.490751 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.492169 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.492760 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.492840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.492895 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.493237 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.493318 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:51:14.493896 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.488069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:40411 (local address 127.4.219.129:39505)
0925 22:51:14.488466 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.488483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.488673 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.489104 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.489107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.489128 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:51:14.489242 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.489249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.490267 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.490270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.491565 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.491575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.492556 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.492560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.492644 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.492665 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.493452 (+ 787us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.493711 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":17,"thread_start_us":115,"threads_started":1}
I20260925 22:51:14.493896 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.488963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:41455 (local address 127.4.219.129:58749)
0925 22:51:14.489326 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.489341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:14.489466 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:14.489791 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:14.489794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:14.489808 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:51:14.489920 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.489925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.490904 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.490907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.492042 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:14.492051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.492979 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.492982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:14.493087 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.493110 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:51:14.493613 (+ 503us) client_negotiation.cc:240] Negotiation successful
0925 22:51:14.493711 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260925 22:51:14.494760 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad5805959d4073aa5bda5fa2823733" candidate_uuid: "f4830106851449c6a0ebb5638c48e15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" is_pre_election: true
I20260925 22:51:14.494771 16617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad5805959d4073aa5bda5fa2823733" candidate_uuid: "f4830106851449c6a0ebb5638c48e15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed346998e600401688328f7b555ee0d9" is_pre_election: true
I20260925 22:51:14.495112 16485 raft_consensus.cc:2454] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4830106851449c6a0ebb5638c48e15a in term 0.
I20260925 22:51:14.495112 16617 raft_consensus.cc:2454] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4830106851449c6a0ebb5638c48e15a in term 0.
I20260925 22:51:14.495620 16295 leader_election.cc:302] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6016d1fad6ad4ecfa604afe4b99a01a1, f4830106851449c6a0ebb5638c48e15a; no voters:
I20260925 22:51:14.495921 16703 raft_consensus.cc:2781] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:51:14.496028 16703 raft_consensus.cc:482] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:51:14.496078 16703 raft_consensus.cc:3037] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:51:14.497475 16703 raft_consensus.cc:504] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.497855 16703 leader_election.cc:288] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [CANDIDATE]: Term 1 election: Requested vote from peers ed346998e600401688328f7b555ee0d9 (127.4.219.131:40411), 6016d1fad6ad4ecfa604afe4b99a01a1 (127.4.219.130:41455)
I20260925 22:51:14.498301 16617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad5805959d4073aa5bda5fa2823733" candidate_uuid: "f4830106851449c6a0ebb5638c48e15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed346998e600401688328f7b555ee0d9"
I20260925 22:51:14.498423 16617 raft_consensus.cc:3037] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:51:14.498963 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bad5805959d4073aa5bda5fa2823733" candidate_uuid: "f4830106851449c6a0ebb5638c48e15a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1"
I20260925 22:51:14.499085 16485 raft_consensus.cc:3037] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:51:14.499634 16617 raft_consensus.cc:2454] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4830106851449c6a0ebb5638c48e15a in term 1.
I20260925 22:51:14.500059 16297 leader_election.cc:302] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed346998e600401688328f7b555ee0d9, f4830106851449c6a0ebb5638c48e15a; no voters:
I20260925 22:51:14.500275 16703 raft_consensus.cc:2781] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:51:14.500278 16485 raft_consensus.cc:2454] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4830106851449c6a0ebb5638c48e15a in term 1.
I20260925 22:51:14.500485 16703 raft_consensus.cc:686] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 1 LEADER]: Becoming Leader. State: Replica: f4830106851449c6a0ebb5638c48e15a, State: Running, Role: LEADER
I20260925 22:51:14.500854 16703 consensus_queue.cc:237] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } }
I20260925 22:51:14.504699 16193 catalog_manager.cc:5495] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a reported cstate change: term changed from 0 to 1, leader changed from <none> to f4830106851449c6a0ebb5638c48e15a (127.4.219.129). New cstate: current_term: 1 leader_uuid: "f4830106851449c6a0ebb5638c48e15a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4830106851449c6a0ebb5638c48e15a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 42611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 } health_report { overall_health: UNKNOWN } } }
I20260925 22:51:14.599387 16692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.595827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.4.219.129:42611)
0925 22:51:14.596001 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:14.596005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:14.596019 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:51:14.596095 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:14.596097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:14.596148 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:14.596240 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:51:14.596552 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.597077 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.598017 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:14.598244 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:14.598323 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:14.598593 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:51:14.599010 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:51:14.599101 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:14.599181 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:51:14.599231 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 22:51:14.606189 16485 raft_consensus.cc:1261] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 1 FOLLOWER]: Refusing update from remote peer f4830106851449c6a0ebb5638c48e15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:51:14.606189 16617 raft_consensus.cc:1261] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 1 FOLLOWER]: Refusing update from remote peer f4830106851449c6a0ebb5638c48e15a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:51:14.606949 16709 consensus_queue.cc:1035] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6016d1fad6ad4ecfa604afe4b99a01a1" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 41455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:51:14.607139 16703 consensus_queue.cc:1035] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed346998e600401688328f7b555ee0d9" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 40411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:51:14.614281 16719 mvcc.cc:205] Tried to move back new op lower bound from 7333382859178573824 to 7333382858759094272. Current Snapshot: MvccSnapshot[applied={T|T < 7333382859178573824}]
I20260925 22:51:14.615391 16721 mvcc.cc:205] Tried to move back new op lower bound from 7333382859178573824 to 7333382858759094272. Current Snapshot: MvccSnapshot[applied={T|T < 7333382859178573824}]
I20260925 22:51:14.626135 16193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:51:14.626328 16193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:51:14.629016 16193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93bf881e349641f897a0dade83d82311: Sending DeleteTablet for 3 replicas of tablet 7bad5805959d4073aa5bda5fa2823733
I20260925 22:51:14.629784 16342 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad5805959d4073aa5bda5fa2823733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:51:14 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:50172
I20260925 22:51:14.630306 16612 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad5805959d4073aa5bda5fa2823733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:51:14 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:46202
I20260925 22:51:14.630383 16729 tablet_replica.cc:330] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a: stopping tablet replica
I20260925 22:51:14.630772 16729 raft_consensus.cc:2229] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:51:14.631229 16730 tablet_replica.cc:330] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: stopping tablet replica
I20260925 22:51:14.631273 16729 raft_consensus.cc:2258] T 7bad5805959d4073aa5bda5fa2823733 P f4830106851449c6a0ebb5638c48e15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:51:14.631516 16477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bad5805959d4073aa5bda5fa2823733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:51:14 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:37534
I20260925 22:51:14.631609 16730 raft_consensus.cc:2229] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:51:14.632169 16731 tablet_replica.cc:330] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: stopping tablet replica
I20260925 22:51:14.632427 16731 raft_consensus.cc:2229] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:51:14.632659 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 16275
I20260925 22:51:14.632784 16731 raft_consensus.cc:2258] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:51:14.634194 16731 ts_tablet_manager.cc:1905] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:51:14.634532 16730 raft_consensus.cc:2258] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:51:14.637583 16730 ts_tablet_manager.cc:1905] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:51:14.637701 16731 ts_tablet_manager.cc:1918] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:51:14.637799 16731 log.cc:1196] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/7bad5805959d4073aa5bda5fa2823733
I20260925 22:51:14.638281 16731 ts_tablet_manager.cc:1939] T 7bad5805959d4073aa5bda5fa2823733 P 6016d1fad6ad4ecfa604afe4b99a01a1: Deleting consensus metadata
I20260925 22:51:14.639678 16178 catalog_manager.cc:4841] TS 6016d1fad6ad4ecfa604afe4b99a01a1 (127.4.219.130:41455): tablet 7bad5805959d4073aa5bda5fa2823733 (table test-table [id=718d5a442cef4c10a9fbc65fb146ea17]) successfully deleted
I20260925 22:51:14.639909 16730 ts_tablet_manager.cc:1918] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:51:14.640025 16730 log.cc:1196] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/7bad5805959d4073aa5bda5fa2823733
I20260925 22:51:14.640287 16730 ts_tablet_manager.cc:1939] T 7bad5805959d4073aa5bda5fa2823733 P ed346998e600401688328f7b555ee0d9: Deleting consensus metadata
I20260925 22:51:14.641144 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 16414
W20260925 22:51:14.641129 16179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:42611 (error 108)
I20260925 22:51:14.642150 16180 catalog_manager.cc:4841] TS ed346998e600401688328f7b555ee0d9 (127.4.219.131:40411): tablet 7bad5805959d4073aa5bda5fa2823733 (table test-table [id=718d5a442cef4c10a9fbc65fb146ea17]) successfully deleted
I20260925 22:51:14.642531 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:14.642146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:42611 (local address 127.0.0.1:50192)
0925 22:51:14.642331 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:14.642420 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:42611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 22:51:14.642688 16179 catalog_manager.cc:4584] TS f4830106851449c6a0ebb5638c48e15a (127.4.219.129:42611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bad5805959d4073aa5bda5fa2823733: Network error: Client connection negotiation failed: client connection to 127.4.219.129:42611: connect: Connection refused (error 111)
I20260925 22:51:14.646816 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 16549
I20260925 22:51:14.652180 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 16162
I20260925 22:51:14.661216 4974 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:51:15.006017 4974 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:51:15.006150 4974 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:51:15.351226 4974 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:51:15.351 UTC [15748] LOG: received smart shutdown request
2026-09-25 22:51:15.354 UTC [15748] LOG: background worker "logical replication launcher" (PID 15757) exited with exit code 1
2026-09-25 22:51:15.354 UTC [15752] LOG: shutting down
2026-09-25 22:51:15.524 UTC [15748] LOG: database system is shut down
2026-09-25T22:51:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47050 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:51:15 dist-test-slave-kb1g krb5kdc[16741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:51:15 dist-test-slave-kb1g krb5kdc[16741](info): set up 2 sockets
Sep 25 22:51:15 dist-test-slave-kb1g krb5kdc[16741](info): commencing operation
krb5kdc: starting...
W20260925 22:51:16.700733 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:51:16 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:51:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangeradmin_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangeradmin_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangerlookup_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangerlookup_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
I20260925 22:51:16.868144 4974 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskre8WLI/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/postgres -l logfile start
2026-09-25 22:51:19.339 UTC [16771] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:51:19.339 UTC [16771] LOG: listening on IPv4 address "127.4.219.148", port 46165
2026-09-25 22:51:19.343 UTC [16771] LOG: listening on Unix socket "/tmp/.s.PGSQL.46165"
2026-09-25 22:51:19.364 UTC [16774] LOG: database system was shut down at 2026-09-25 22:51:17 UTC
2026-09-25 22:51:19.368 UTC [16771] LOG: database system is ready to accept connections
I20260925 22:51:20.372968 4974 mini_postgres.cc:96] Postgres bound to 46165
2026-09-25 22:51:20.379 UTC [16782] FATAL: database "slave" does not exist
127.4.219.148:46165 - accepting connections
I20260925 22:51:20.380192 4974 mini_ranger.cc:161] Starting Ranger...
I20260925 22:51:20.394968 4974 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:51:20.877313 4974 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:51:20.877382 4974 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin
2026-09-25 22:51:21,174 [I] DB FLAVOR :POSTGRES
2026-09-25 22:51:21,174 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:51:21,174 [I] Checking connection..
2026-09-25 22:51:21,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:51:21,410 [I] Checking connection passed.
2026-09-25 22:51:21,411 [I] --------- Verifying version history table ---------
2026-09-25 22:51:21,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:51:21,642 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:51:21,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:51:21,891 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:51:21,891 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:51:21,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:51:22.181 UTC [16846] WARNING: there is no transaction in progress
2026-09-25 22:51:22,188 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:51:22,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:51:22,448 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:51:22,448 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:51:22,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:51:22,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:51:22,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kb1g.c.gcp-upstream.internal','N') ;"
2026-09-25 22:51:23,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:51:23,482 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:51:23,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:51:23,728 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:51:23,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:51:23,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:51:24,184 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:51:24,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:51:25.149 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.163 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.180 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.196 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.215 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.233 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.288 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.296 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.311 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.321 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.332 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.348 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.357 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.364 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.371 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:25.378 UTC [16954] WARNING: there is no transaction in progress
2026-09-25 22:51:26,000 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:51:26,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:51:26,238 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:51:26,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:51:26,504 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:51:26,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:51:26,735 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:51:26,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:51:26,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:51:27,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kb1g.c.gcp-upstream.internal';"
2026-09-25 22:51:27,431 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:51:27,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:51:27,655 [I] DB_PATCHES have already been applied
I20260925 22:51:27.661100 4974 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskre8WLI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:51:27.661166 4974 mini_ranger.cc:193] Using host: 127.4.219.148
I20260925 22:51:27.664639 4974 mini_ranger.cc:227] Ranger admin URL: http://127.4.219.148:48349
Sep 25, 2026 10:51:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:51:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:51:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:51:28.768585 4974 mini_ranger.cc:160] Time spent starting Ranger: real 8.388s user 0.000s sys 0.010s
Sep 25, 2026 10:51:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:51:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 10:51:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:51:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:51:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:51:49.489465 4974 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.219.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.219.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangerkms_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.219.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/rangerkms_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.219.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.219.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.219.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/HTTP_127.4.219.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:51:49 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:51:49.574735 4974 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:51:49.583994 4974 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:51:50.453085 4974 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:51:50.453200 4974 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms
I20260925 22:51:50.453225 4974 mini_ranger_kms.cc:227] Using postgres at 127.4.219.148:46165
2026-09-25 22:51:50,628 [I] DB FLAVOR :POSTGRES
2026-09-25 22:51:50,628 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:51:50,628 [I] Checking connection
2026-09-25 22:51:50,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:51:50,872 [I] connection success
2026-09-25 22:51:50,872 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:51:50,872 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:51:50,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:51:51,101 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:51:51,102 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:51:51,102 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:51:51,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.219.148:46165/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:51:51,365 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:51:51.804716 4974 mini_ranger_kms.cc:310] Created kms service
I20260925 22:51:51.999318 4974 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:51:52.044571 4974 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:51:52 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376709, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:52.473505 4974 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:51:52.473760 4974 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms:/tmp/dist-test-taskre8WLI/build/debug/bin/postgresql.jar:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskre8WLI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:51:52.473786 4974 mini_ranger_kms.cc:240] Using host: 127.4.219.148
I20260925 22:51:52.477522 4974 mini_ranger_kms.cc:276] Ranger KMS PID: 17134
I20260925 22:51:52.477598 4974 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.219.148:60023
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:51:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:51:54.668856 4974 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.094s user 0.018s sys 0.141s
I20260925 22:51:54.668987 4974 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:51:54.669075 4974 mini_ranger_kms.cc:401] 127.4.219.148:60023/kms/v1/keys
Sep 25 22:51:54 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:51:55 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
Sep 25 22:51:55 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376714, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.219.148@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
Sep 25 22:51:57 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:51:57.821807 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41107
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41107
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:57.960516 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:57.960871 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:57.960942 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:57.964970 17185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:51:57.965044 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:57.965062 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:57.965081 17185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:51:57.965096 17185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:51:57.970800 17185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.219.148:60023/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41107
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:57.971930 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:57.973012 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:57.982864 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:57.982873 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:57.982873 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:57.983031 17185 server_base.cc:1034] running on GCE node
I20260925 22:51:57.983906 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:57.985060 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:57.986236 17185 hybrid_clock.cc:648] HybridClock initialized: now 1790376717986218 us; error 34 us; skew 500 ppm
Sep 25 22:51:57 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376717, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:57.993837 17185 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:51:57.994922 17185 webserver.cc:460] Webserver started at http://127.4.219.190:43531/ using document root <none> and password file <none>
I20260925 22:51:57.995472 17185 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:57.995534 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:57.995751 17185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:57 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376717, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:58.042610 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "81b160f9b6274fa3a56337a11c99d9fb"
format_stamp: "Formatted at 2026-09-25 22:51:58 on dist-test-slave-kb1g"
server_key: "a40f517c791eacf6ab3e3a57d7ac7634"
server_key_iv: "fa69f371ffd25d9a2ebb6650a38eeedc"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:58.043246 17185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "81b160f9b6274fa3a56337a11c99d9fb"
format_stamp: "Formatted at 2026-09-25 22:51:58 on dist-test-slave-kb1g"
server_key: "a40f517c791eacf6ab3e3a57d7ac7634"
server_key_iv: "fa69f371ffd25d9a2ebb6650a38eeedc"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:58.047437 17185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:51:58.062060 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:58.063555 17185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:51:58.063715 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "81b160f9b6274fa3a56337a11c99d9fb"
format_stamp: "Formatted at 2026-09-25 22:51:58 on dist-test-slave-kb1g"
server_key: "a40f517c791eacf6ab3e3a57d7ac7634"
server_key_iv: "fa69f371ffd25d9a2ebb6650a38eeedc"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:58.063865 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:58.087241 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:58.090408 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:58.090621 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:58.099001 17185 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41107
I20260925 22:51:58.099001 17252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41107 every 8 connection(s)
I20260925 22:51:58.100144 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:51:58.103055 17253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:51:58.108660 17253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Bootstrap starting.
I20260925 22:51:58.110210 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17185
I20260925 22:51:58.110325 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:51:58.110594 4974 external_mini_cluster.cc:1428] Setting key 8e257b56533486dc8114107dfd865c1e
I20260925 22:51:58.111420 17253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Neither blocks nor log segments found. Creating new log.
I20260925 22:51:58.112366 17253 log.cc:824] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:51:58.114598 17253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: No bootstrap required, opened a new log
I20260925 22:51:58.118827 17253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:51:58.119125 17253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:51:58.119205 17253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b160f9b6274fa3a56337a11c99d9fb, State: Initialized, Role: FOLLOWER
I20260925 22:51:58.119758 17253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:51:58.119941 17253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:51:58.120030 17253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:51:58.120151 17253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 0 FOLLOWER]: Advancing to term 1
Sep 25 22:51:58 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:58.121183 17253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:51:58.121562 17253 leader_election.cc:302] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b160f9b6274fa3a56337a11c99d9fb; no voters:
I20260925 22:51:58.121934 17253 leader_election.cc:288] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:51:58.122056 17258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:51:58.122300 17258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 LEADER]: Becoming Leader. State: Replica: 81b160f9b6274fa3a56337a11c99d9fb, State: Running, Role: LEADER
I20260925 22:51:58.122644 17258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:51:58.123474 17253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:51:58.123880 17258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b160f9b6274fa3a56337a11c99d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } } }
I20260925 22:51:58.123994 17258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: This master's current role is: LEADER
I20260925 22:51:58.124186 17258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b160f9b6274fa3a56337a11c99d9fb. Latest consensus state: current_term: 1 leader_uuid: "81b160f9b6274fa3a56337a11c99d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } } }
I20260925 22:51:58.124260 17258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: This master's current role is: LEADER
I20260925 22:51:58.125880 17253 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:51:58.126732 17262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:51:58.128372 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:58.111753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.4.219.190:41107)
0925 22:51:58.112310 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:58.112329 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:58.112372 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:51:58.113176 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:58.113202 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:58.113622 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:58.113987 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:58.114816 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:58.116405 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:58.117668 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:58.118474 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:58.118632 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:58.121908 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:58.121915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:58.121933 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:58.121968 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:58.124704 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:58.125320 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:58.125329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:58.125338 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:58.125480 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:58.125837 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:58.125843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:58.125848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:58.126421 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:58.127305 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:58.127430 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 22:51:58.127651 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":124,"threads_started":1}
I20260925 22:51:58.130955 17262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:51:58.141916 17262 catalog_manager.cc:1347] Generated new cluster ID: f403adaac2d943ac86d0c4393516e190
I20260925 22:51:58.142006 17262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:51:58.152750 17262 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:51:58.154078 17262 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:51:58.160645 17262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Generated new TSK 0
I20260925 22:51:58.161509 17262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:51:58.899859 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:59.046449 17298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:59.046720 17298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:59.046761 17298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:59.051851 17298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:59.051934 17298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:59.052019 17298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:51:59.057780 17298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:59.058888 17298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:59.059988 17298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:59.071739 17306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:59.071817 17308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:59.072121 17305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:59.073835 17298 server_base.cc:1034] running on GCE node
I20260925 22:51:59.074381 17298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:59.075073 17298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:59.076244 17298 hybrid_clock.cc:648] HybridClock initialized: now 1790376719076217 us; error 44 us; skew 500 ppm
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:59.079272 17298 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:51:59.080428 17298 webserver.cc:460] Webserver started at http://127.4.219.129:33007/ using document root <none> and password file <none>
I20260925 22:51:59.081066 17298 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:59.081163 17298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:59.081390 17298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:59.097836 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "aa8583aa29df467c85f6341d3101de2d"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "538c05672e733d034007ed0d77bac549"
server_key_iv: "f844d8c3979a62572bcee1b6f7bcbb47"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.098644 17298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "aa8583aa29df467c85f6341d3101de2d"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "538c05672e733d034007ed0d77bac549"
server_key_iv: "f844d8c3979a62572bcee1b6f7bcbb47"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.104097 17298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 22:51:59.128971 17315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.130791 17298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 22:51:59.130914 17298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "aa8583aa29df467c85f6341d3101de2d"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "538c05672e733d034007ed0d77bac549"
server_key_iv: "f844d8c3979a62572bcee1b6f7bcbb47"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.131050 17298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:59.176144 17298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:59.180186 17298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:59.180471 17298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:59.181324 17298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:59.182511 17298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:59.182641 17298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.182785 17298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:59.182885 17298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.197429 17298 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:41321
I20260925 22:51:59.198722 17298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:51:59.200410 17430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:41321 every 8 connection(s)
I20260925 22:51:59.201900 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17298
I20260925 22:51:59.202018 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:51:59.202304 4974 external_mini_cluster.cc:1428] Setting key 79a62f4d045917296a2dc7275d90ef63
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:51:59.217782 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.201033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:50791 (local address 127.4.219.190:41107)
0925 22:51:59.201336 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:59.201342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:59.202218 (+ 876us) server_negotiation.cc:407] Connection header received
0925 22:51:59.203555 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:59.203560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:59.203615 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:59.203681 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:59.205408 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.206399 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.208263 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.208659 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.208771 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.212183 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:59.212190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:59.212197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:59.212237 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:59.214527 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.215262 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.215268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.215271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.215374 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.215819 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.215825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.215829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.216110 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:59.216237 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:59.217464 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0925 22:51:59.217600 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260925 22:51:59.218473 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.201474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.129:50791)
0925 22:51:59.202043 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:59.202108 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:59.203293 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:59.204117 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:59.204127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:59.204496 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:59.205231 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.205251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.206565 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.206568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.208122 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.208133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.208946 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.208950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.209145 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.209938 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:59.209961 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:59.211966 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:59.214675 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.214684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.214704 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.215124 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.215489 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.215493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.215496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.215692 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.216319 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:59.216330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:59.216710 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 22:51:59.216937 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 22:51:59.217343 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":160,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:51:59.220052 17431 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:51:59.220316 17431 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:59.220989 17431 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:51:59.222921 17217 ts_manager.cc:194] Registered new tserver with Master: aa8583aa29df467c85f6341d3101de2d (127.4.219.129:41321)
I20260925 22:51:59.224817 17217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:50791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:51:59.239566 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:51:59.373396 17437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:59.373759 17437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:59.373827 17437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:59.378631 17437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:59.378698 17437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:59.378759 17437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:51:59.385313 17437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:59.386658 17437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:59.387964 17437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:59.399834 17443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:59.400123 17444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:59.400422 17437 server_base.cc:1034] running on GCE node
W20260925 22:51:59.399837 17446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:59.400969 17437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:59.401708 17437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:59.402894 17437 hybrid_clock.cc:648] HybridClock initialized: now 1790376719402872 us; error 43 us; skew 500 ppm
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:59.405676 17437 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:51:59.406589 17437 webserver.cc:460] Webserver started at http://127.4.219.130:33947/ using document root <none> and password file <none>
I20260925 22:51:59.407192 17437 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:59.407249 17437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:59.407404 17437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:59.421000 17437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "c49f8ac6e6ab47d2806148c36fa647ef"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "1e8a1f16d9d40d54d8da7d664f7a82fb"
server_key_iv: "98be7991ecf69d9db824a1532e8c1e76"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.421561 17437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "c49f8ac6e6ab47d2806148c36fa647ef"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "1e8a1f16d9d40d54d8da7d664f7a82fb"
server_key_iv: "98be7991ecf69d9db824a1532e8c1e76"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.426298 17437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:51:59.444926 17453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.450454 17437 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260925 22:51:59.450608 17437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "c49f8ac6e6ab47d2806148c36fa647ef"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "1e8a1f16d9d40d54d8da7d664f7a82fb"
server_key_iv: "98be7991ecf69d9db824a1532e8c1e76"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.450765 17437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:59.483192 17437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:59.486970 17437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:59.487205 17437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:59.487998 17437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:59.489159 17437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:59.489274 17437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.489377 17437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:59.489444 17437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.499666 17437 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:45565
I20260925 22:51:59.500285 17566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:45565 every 8 connection(s)
I20260925 22:51:59.500859 17437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:51:59.501991 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17437
I20260925 22:51:59.502138 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:51:59.502476 4974 external_mini_cluster.cc:1428] Setting key 34a0353cf3fe277ef2f0574c6550a8d1
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:59.526782 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.502953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:34797 (local address 127.4.219.190:41107)
0925 22:51:59.503084 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:59.503089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:59.507276 (+ 4187us) server_negotiation.cc:407] Connection header received
0925 22:51:59.508645 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:59.508649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:59.508705 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:59.508806 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:59.510464 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.511412 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.513191 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.513585 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.513673 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.521851 (+ 8178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:59.521856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:59.521859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:59.521891 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:59.523876 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.524658 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.524662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.524664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.524779 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.525263 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.525267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.525269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.525544 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:59.525644 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:59.526421 (+ 777us) server_negotiation.cc:299] Negotiation successful
0925 22:51:59.526595 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:51:59.528738 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.503491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.130:34797)
0925 22:51:59.507084 (+ 3593us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:59.507147 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:59.508294 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:59.509123 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:59.509134 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:59.509505 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:59.510277 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.510298 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.511563 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.511567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.513043 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.513054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.514254 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.514259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.514483 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.519522 (+ 5039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:59.519549 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:59.521585 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:59.524011 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.524021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.524042 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.524473 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.524927 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.524930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.524932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.525138 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.525721 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:59.525729 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:59.526106 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 22:51:59.527059 (+ 953us) client_negotiation.cc:240] Negotiation successful
0925 22:51:59.527482 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3338,"thread_start_us":156,"threads_started":1}
I20260925 22:51:59.530164 17567 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:51:59.530448 17567 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:59.531111 17567 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:51:59.532269 17217 ts_manager.cc:194] Registered new tserver with Master: c49f8ac6e6ab47d2806148c36fa647ef (127.4.219.130:45565)
I20260925 22:51:59.533135 17217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:34797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:51:59.547813 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:51:59.691495 17572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:51:59.691785 17572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:51:59.691867 17572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:51:59.695983 17572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:51:59.696065 17572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:51:59.696178 17572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:51:59.701987 17572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:51:59.703058 17572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:51:59.704257 17572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:51:59.714879 17578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:59.715309 17580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:51:59.716101 17577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:51:59.717101 17572 server_base.cc:1034] running on GCE node
I20260925 22:51:59.717567 17572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:51:59.718320 17572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:51:59.719651 17572 hybrid_clock.cc:648] HybridClock initialized: now 1790376719719570 us; error 94 us; skew 500 ppm
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:51:59.723110 17572 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:51:59.724255 17572 webserver.cc:460] Webserver started at http://127.4.219.131:46067/ using document root <none> and password file <none>
I20260925 22:51:59.725024 17572 fs_manager.cc:359] Metadata directory not provided
I20260925 22:51:59.725133 17572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:51:59.725375 17572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:51:59.753997 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "2a1f7ad1bb71488eace00748e715d9df"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "2e49d21c328f954d0e118a7d3a4dc7db"
server_key_iv: "43364d0cb0b88ec69eaeba86612943a0"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.754802 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "2a1f7ad1bb71488eace00748e715d9df"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "2e49d21c328f954d0e118a7d3a4dc7db"
server_key_iv: "43364d0cb0b88ec69eaeba86612943a0"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.761023 17572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 22:51:59.794081 17587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.795822 17572 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260925 22:51:59.795967 17572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "2a1f7ad1bb71488eace00748e715d9df"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "2e49d21c328f954d0e118a7d3a4dc7db"
server_key_iv: "43364d0cb0b88ec69eaeba86612943a0"
server_key_version: "kuduclusterkey@0"
I20260925 22:51:59.796123 17572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:51:59.819686 17572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:51:59.823382 17572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:51:59.823583 17572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:51:59.824322 17572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:51:59.825479 17572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:51:59.825541 17572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.825591 17572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:51:59.825618 17572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:51:59.836226 17572 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:43165
I20260925 22:51:59.836313 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:43165 every 8 connection(s)
I20260925 22:51:59.837373 17572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:51:59.839484 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17572
I20260925 22:51:59.839605 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:51:59.839876 4974 external_mini_cluster.cc:1428] Setting key 0463f83618a5bf67243ba0571067edf1
Sep 25 22:51:59 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376719, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:51:59.853345 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.839960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.131:37175)
0925 22:51:59.841813 (+ 1853us) negotiation.cc:107] Waiting for socket to connect
0925 22:51:59.841875 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:51:59.842747 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:51:59.843290 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:51:59.843297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:51:59.843568 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:51:59.844043 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.844056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.844908 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.844911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.845910 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:51:59.845920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.846405 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.846409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:51:59.846581 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.847196 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:51:59.847211 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:51:59.848611 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:51:59.850349 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.850356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.850371 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.850651 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.850937 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:51:59.850940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:51:59.850943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:51:59.851103 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:51:59.851503 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:51:59.851510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:51:59.851754 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 22:51:59.851961 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:51:59.852273 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"thread_start_us":149,"threads_started":1}
I20260925 22:51:59.854039 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:51:59.839468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:37175 (local address 127.4.219.190:41107)
0925 22:51:59.839598 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:51:59.839603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:51:59.841961 (+ 2358us) server_negotiation.cc:407] Connection header received
0925 22:51:59.842957 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:51:59.842961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:51:59.843016 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:51:59.843095 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:51:59.844179 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.844787 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.846036 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:51:59.846263 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:51:59.846374 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:51:59.848766 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:51:59.848769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:51:59.848772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:51:59.848793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:51:59.850220 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.850755 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.850760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.850762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.850840 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:51:59.851199 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:51:59.851204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:51:59.851206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:51:59.851382 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:51:59.851504 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:51:59.851993 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 22:51:59.853917 (+ 1924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:51:59.854803 17701 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:51:59.855036 17701 heartbeater.cc:464] Registering TS with master...
I20260925 22:51:59.855526 17701 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:51:59.856618 17217 ts_manager.cc:194] Registered new tserver with Master: 2a1f7ad1bb71488eace00748e715d9df (127.4.219.131:43165)
I20260925 22:51:59.857306 17217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:37175
I20260925 22:51:59.865958 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:51:59.866792 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17298
I20260925 22:51:59.873793 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17437
I20260925 22:51:59.880146 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17572
I20260925 22:51:59.886979 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17185
I20260925 22:51:59.893725 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41107
--webserver_interface=127.4.219.190
--webserver_port=43531
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41107
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:00.006904 17704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:00.007185 17704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:00.007261 17704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:00.011353 17704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:00.011411 17704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:00.011431 17704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:00.011447 17704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:00.011512 17704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:00.017007 17704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41107
--ranger_config_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=43531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:00.018138 17704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:00.019212 17704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:00.027987 17710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.028158 17704 server_base.cc:1034] running on GCE node
W20260925 22:52:00.027971 17712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:00.027971 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.028865 17704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:00.030014 17704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:00.031167 17704 hybrid_clock.cc:648] HybridClock initialized: now 1790376720031137 us; error 43 us; skew 500 ppm
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:00.034016 17704 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:00.034904 17704 webserver.cc:460] Webserver started at http://127.4.219.190:43531/ using document root <none> and password file <none>
I20260925 22:52:00.035441 17704 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:00.035535 17704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:00.103543 17704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:00.106091 17720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.107502 17704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:52:00.107656 17704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "81b160f9b6274fa3a56337a11c99d9fb"
format_stamp: "Formatted at 2026-09-25 22:51:58 on dist-test-slave-kb1g"
server_key: "a40f517c791eacf6ab3e3a57d7ac7634"
server_key_iv: "fa69f371ffd25d9a2ebb6650a38eeedc"
server_key_version: "kuduclusterkey@0"
I20260925 22:52:00.108302 17704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:00.131610 17704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:00.134809 17704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:00.135030 17704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:00.142558 17704 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41107
I20260925 22:52:00.142580 17772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41107 every 8 connection(s)
I20260925 22:52:00.143630 17704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:00.148130 17773 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:52:00.149971 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Bootstrap starting.
I20260925 22:52:00.151314 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17704
I20260925 22:52:00.151752 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:41321
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:41107
--webserver_port=33007
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:00.168118 17773 log.cc:824] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:00.175443 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:00.175894 17773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Bootstrap complete.
I20260925 22:52:00.180763 17773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:52:00.181185 17773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b160f9b6274fa3a56337a11c99d9fb, State: Initialized, Role: FOLLOWER
I20260925 22:52:00.181725 17773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:52:00.181847 17773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:00.181943 17773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:00.182051 17773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:52:00.184090 17773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:52:00.184448 17773 leader_election.cc:302] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b160f9b6274fa3a56337a11c99d9fb; no voters:
I20260925 22:52:00.184877 17773 leader_election.cc:288] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:52:00.185000 17777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:52:00.185281 17777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [term 2 LEADER]: Becoming Leader. State: Replica: 81b160f9b6274fa3a56337a11c99d9fb, State: Running, Role: LEADER
I20260925 22:52:00.185592 17777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } }
I20260925 22:52:00.186228 17773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:00.187054 17778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81b160f9b6274fa3a56337a11c99d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } } }
I20260925 22:52:00.187206 17778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:00.187508 17779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b160f9b6274fa3a56337a11c99d9fb. Latest consensus state: current_term: 2 leader_uuid: "81b160f9b6274fa3a56337a11c99d9fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b160f9b6274fa3a56337a11c99d9fb" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41107 } } }
I20260925 22:52:00.187620 17779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:00.187894 17785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:00.191367 17785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:00.191979 17773 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:52:00.192071 17785 catalog_manager.cc:1259] Loaded cluster ID: f403adaac2d943ac86d0c4393516e190
I20260925 22:52:00.192178 17785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:00.194808 17785 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:00.195529 17785 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Loaded TSK: 0
I20260925 22:52:00.196866 17785 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:52:00.292009 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:00.292326 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:00.292407 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:00.297022 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:00.297106 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:00.297209 17775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:00.302634 17775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:41321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=33007
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:00.303694 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:00.305059 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:00.317348 17810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:00.317324 17811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.317566 17775 server_base.cc:1034] running on GCE node
W20260925 22:52:00.317695 17813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.318467 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:00.319145 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:00.320322 17775 hybrid_clock.cc:648] HybridClock initialized: now 1790376720320292 us; error 41 us; skew 500 ppm
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:00.323096 17775 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:00.324086 17775 webserver.cc:460] Webserver started at http://127.4.219.129:33007/ using document root <none> and password file <none>
I20260925 22:52:00.324695 17775 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:00.324800 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:00.353412 17775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:52:00.357118 17821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.358696 17775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 22:52:00.358812 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "aa8583aa29df467c85f6341d3101de2d"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "538c05672e733d034007ed0d77bac549"
server_key_iv: "f844d8c3979a62572bcee1b6f7bcbb47"
server_key_version: "kuduclusterkey@0"
I20260925 22:52:00.359577 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:00.379218 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:00.382359 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:00.382570 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:00.383227 17775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:00.384145 17775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:00.384212 17775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.384281 17775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:00.384335 17775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.395306 17775 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:41321
I20260925 22:52:00.395584 17934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:41321 every 8 connection(s)
I20260925 22:52:00.396394 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:00.405352 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17775
I20260925 22:52:00.405849 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:45565
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:41107
--webserver_port=33947
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:00.423084 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.398889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.129:52699)
0925 22:52:00.399368 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:00.399420 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:00.400343 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:00.401589 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:00.401597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:00.401916 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:00.402423 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.402438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.403736 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.403739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.404654 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.404663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.405765 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.405772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.405962 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.406697 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:00.406722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:00.408594 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:00.411157 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.411163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.411177 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.411598 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.412074 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.412077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.412079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.412256 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.413264 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:00.413270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:00.413588 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 22:52:00.413784 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:52:00.414122 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260925 22:52:00.425921 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.398724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52699 (local address 127.4.219.190:41107)
0925 22:52:00.399925 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:00.399945 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:00.400013 (+ 68us) server_negotiation.cc:407] Connection header received
0925 22:52:00.400876 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:00.400888 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:00.401182 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:00.401491 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:00.402600 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.403598 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.404769 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.405395 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.405549 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.408791 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:00.408799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:00.408815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:00.408847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:00.411017 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.411761 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.411769 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.411779 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.411937 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.412495 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.412499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.412501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.413020 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:00.413181 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:00.420264 (+ 7083us) server_negotiation.cc:299] Negotiation successful
0925 22:52:00.424817 (+ 4553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":104,"threads_started":1}
I20260925 22:52:00.428829 17935 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:52:00.429081 17935 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.429651 17935 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.431949 17935 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:41107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:52:00.432085 17935 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.432256 17935 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:00.433506 17935 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.433678 17935 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.434440 17935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:52:00.549681 17939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:00.549985 17939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:00.550055 17939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:00.554440 17939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:00.554536 17939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:00.554643 17939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:00.560262 17939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:45565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=33947
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:00.561549 17939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:00.562789 17939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:00.573252 17944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:00.573791 17947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:00.574074 17945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.575215 17939 server_base.cc:1034] running on GCE node
I20260925 22:52:00.575697 17939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:00.576498 17939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:00.579823 17939 hybrid_clock.cc:648] HybridClock initialized: now 1790376720579775 us; error 56 us; skew 500 ppm
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:00.583392 17939 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:00.584443 17939 webserver.cc:460] Webserver started at http://127.4.219.130:33947/ using document root <none> and password file <none>
I20260925 22:52:00.585109 17939 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:00.585214 17939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:00.613260 17939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:52:00.616626 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.618140 17939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:00.618240 17939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "c49f8ac6e6ab47d2806148c36fa647ef"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "1e8a1f16d9d40d54d8da7d664f7a82fb"
server_key_iv: "98be7991ecf69d9db824a1532e8c1e76"
server_key_version: "kuduclusterkey@0"
I20260925 22:52:00.618978 17939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:00.662710 17939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:00.666323 17939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:00.666568 17939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:00.667321 17939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:00.668365 17939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:00.668429 17939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.668483 17939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:00.668515 17939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.680011 17939 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:45565
I20260925 22:52:00.680406 18068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:45565 every 8 connection(s)
I20260925 22:52:00.681373 17939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:00.686358 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 17939
I20260925 22:52:00.686834 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.219.148:60023/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:43165
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:41107
--webserver_port=46067
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:00.701875 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.683603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:45787 (local address 127.4.219.190:41107)
0925 22:52:00.683752 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:00.683757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:00.684848 (+ 1091us) server_negotiation.cc:407] Connection header received
0925 22:52:00.688307 (+ 3459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:00.688311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:00.688375 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:00.688436 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:00.690486 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.691402 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.693354 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.693748 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.693834 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.697245 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:00.697249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:00.697252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:00.697284 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:00.699249 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.699907 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.699923 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.699926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.700018 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.700456 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.700460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.700462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.700741 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:00.700833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:00.701544 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 22:52:00.701687 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:52:00.703820 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.684083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.130:45787)
0925 22:52:00.684675 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:00.684734 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:00.688006 (+ 3272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:00.689005 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:00.689017 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:00.689401 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:00.690269 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.690291 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.691585 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.691589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.693197 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.693210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.694067 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.694071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.694284 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.695050 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:00.695073 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:00.697032 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:00.699386 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.699393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.699411 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.699780 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.700132 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.700135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.700137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.700316 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.700905 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:00.700924 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:00.701250 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 22:52:00.702366 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0925 22:52:00.702729 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":151,"threads_started":1}
I20260925 22:52:00.705472 18069 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:52:00.705739 18069 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.706338 18069 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.708030 18069 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:41107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:52:00.708150 18069 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.708307 18069 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:00.709168 18069 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.709329 18069 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.710091 18069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:52:00.838466 18072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:00.838780 18072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:00.838874 18072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:00.843261 18072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:00.843346 18072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:00.843433 18072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:00.850710 18072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.219.148:60023/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:43165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=46067
--tserver_master_addrs=127.4.219.190:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:00.852097 18072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:00.853411 18072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:00.863741 18079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.864120 18072 server_base.cc:1034] running on GCE node
W20260925 22:52:00.863723 18078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:00.863775 18081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:00.864849 18072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:00.865567 18072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:00.866745 18072 hybrid_clock.cc:648] HybridClock initialized: now 1790376720866717 us; error 40 us; skew 500 ppm
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:00.870234 18072 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:00.871294 18072 webserver.cc:460] Webserver started at http://127.4.219.131:46067/ using document root <none> and password file <none>
I20260925 22:52:00.871976 18072 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:00.872095 18072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:00.892755 18072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:52:00.895408 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.896914 18072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:00.897045 18072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "2a1f7ad1bb71488eace00748e715d9df"
format_stamp: "Formatted at 2026-09-25 22:51:59 on dist-test-slave-kb1g"
server_key: "2e49d21c328f954d0e118a7d3a4dc7db"
server_key_iv: "43364d0cb0b88ec69eaeba86612943a0"
server_key_version: "kuduclusterkey@0"
I20260925 22:52:00.897790 18072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:00.911544 18072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:00.914690 18072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:00.914882 18072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:00.915509 18072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:00.916416 18072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:00.916486 18072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.916594 18072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:00.916640 18072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:00.927721 18072 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:43165
I20260925 22:52:00.927819 18202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:43165 every 8 connection(s)
I20260925 22:52:00.928951 18072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:00.936286 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18072
Sep 25 22:52:00 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376720, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:00.948022 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.931549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:41107 (local address 127.4.219.131:42037)
0925 22:52:00.933624 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:00.933683 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:00.934770 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:00.935422 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:00.935432 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:00.935790 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:00.936487 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.936508 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.937684 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.937688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.939122 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:00.939133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.939916 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.939921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:00.940123 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.940849 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:00.940870 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:00.942572 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:00.944628 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.944634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.944651 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.944955 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.945296 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:00.945300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:00.945302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:00.945512 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:00.945995 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:00.946002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:00.946332 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 22:52:00.946550 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:52:00.946927 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1834,"thread_start_us":133,"threads_started":1}
I20260925 22:52:00.948799 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.933419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:42037 (local address 127.4.219.190:41107)
0925 22:52:00.933553 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:00.933557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:00.933804 (+ 247us) server_negotiation.cc:407] Connection header received
0925 22:52:00.935047 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:00.935051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:00.935109 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:00.935205 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:00.936696 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.937517 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.939268 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.939614 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.939696 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.942772 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:00.942777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:00.942780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:00.942805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:00.944463 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.945090 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.945094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.945096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.945185 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.945644 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.945647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.945649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.945867 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:00.945973 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:00.948475 (+ 2502us) server_negotiation.cc:299] Negotiation successful
0925 22:52:00.948643 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:52:00.950910 18203 heartbeater.cc:347] Connected to a master server at 127.4.219.190:41107
I20260925 22:52:00.951176 18203 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.951752 18203 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.953437 18203 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:41107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:52:00.953567 18203 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.953728 18203 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:00.954622 18203 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:00.954730 18206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:00.938606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.4.219.190:41107)
0925 22:52:00.941890 (+ 3284us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:00.941914 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:00.941928 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:52:00.941970 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:00.941975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:00.942023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:00.942115 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:00.942930 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.943731 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.947132 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:00.947481 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:00.947613 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:00.950819 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:00.950823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:00.950825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:00.950853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:00.952420 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.953025 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.953028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.953030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.953115 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:00.953401 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:00.953404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:00.953406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:00.953649 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:00.953868 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:00.954209 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:52:00.954543 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3193,"thread_start_us":106,"threads_started":1}
I20260925 22:52:00.955005 18203 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
W20260925 22:52:00.955914 18203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 22:52:01 dist-test-slave-kb1g krb5kdc[16741](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376721, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:01.434686 17935 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:01.435021 17935 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:01.436407 17731 ts_manager.cc:194] Registered new tserver with Master: aa8583aa29df467c85f6341d3101de2d (127.4.219.129:41321)
I20260925 22:52:01.438196 17731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:52699
Sep 25 22:52:01 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376721, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:01.711084 18069 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:01.711419 18069 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:01.712218 17737 ts_manager.cc:194] Registered new tserver with Master: c49f8ac6e6ab47d2806148c36fa647ef (127.4.219.130:45565)
I20260925 22:52:01.713069 17737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:45787
I20260925 22:52:01.956149 18203 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:01.956454 18203 heartbeater.cc:511] Master 127.4.219.190:41107 requested a full tablet report, sending...
I20260925 22:52:01.957255 17731 ts_manager.cc:194] Registered new tserver with Master: 2a1f7ad1bb71488eace00748e715d9df (127.4.219.131:43165)
I20260925 22:52:01.957864 17731 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:42037
I20260925 22:52:01.959235 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:52:01 dist-test-slave-kb1g krb5kdc[16741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376721, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for HTTP/127.4.219.148@KRBTEST.COM
I20260925 22:52:01.975461 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:01.960874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.4.219.190:41107)
0925 22:52:01.961122 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:01.961127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:01.961245 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:52:01.961416 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:01.961420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:01.961467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:01.961570 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:01.962276 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:01.962901 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:01.964401 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:01.964782 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:01.964962 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:01.970855 (+ 5893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:01.970859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:01.970862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:01.970891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:01.972986 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:01.973478 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:01.973482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:01.973484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:01.973583 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:01.973871 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:01.973874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:01.973876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:01.974096 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:01.974225 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:01.975003 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 22:52:01.975282 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260925 22:52:01.979722 17737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:52:01.984714 17737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:52:02.018244 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.006550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:43165 (local address 127.0.0.1:42092)
0925 22:52:02.011093 (+ 4543us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.011145 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.011293 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:02.012175 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:02.012181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:02.012204 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:02.012378 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.012385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.014910 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.014913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.016233 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.016244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.017495 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.017499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.017809 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.017849 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:52:02.017959 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:52:02.018082 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3658,"thread_start_us":100,"threads_started":1}
I20260925 22:52:02.019934 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.007329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.4.219.131:43165)
0925 22:52:02.011573 (+ 4244us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.011581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.011603 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:52:02.011677 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.011682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.011944 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.012552 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:02.012683 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.014792 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.016367 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.017180 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.017278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.017426 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.019689 (+ 2263us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.019795 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4125,"thread_start_us":89,"threads_started":1}
I20260925 22:52:02.023195 18137 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e (DEFAULT_TABLE table=test-table [id=af29fe6bc40c4657b13c709424a90457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:02.024386 18137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbdfcf9efa45e69823b8e021046d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:02.026073 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.009811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41321 (local address 127.0.0.1:56420)
0925 22:52:02.018611 (+ 8800us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.018625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.018687 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:02.021619 (+ 2932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:02.021622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:02.021638 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:02.021710 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.021716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.023102 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.023106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.024395 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.024406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.025705 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.025708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.025800 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.025825 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:52:02.025871 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 22:52:02.025937 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8747,"thread_start_us":72,"threads_started":1}
I20260925 22:52:02.026778 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.009982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.4.219.129:41321)
0925 22:52:02.021033 (+ 11051us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.021040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.021065 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:52:02.021137 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.021142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.021320 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.021458 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:02.021835 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.022968 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.024683 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.025570 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.026336 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.026480 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.026528 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.026642 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10940,"thread_start_us":96,"threads_started":1}
I20260925 22:52:02.029877 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e (DEFAULT_TABLE table=test-table [id=af29fe6bc40c4657b13c709424a90457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:02.031040 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbdfcf9efa45e69823b8e021046d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:02.041934 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.008776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:45565 (local address 127.0.0.1:56256)
0925 22:52:02.014129 (+ 5353us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.014142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.014225 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:02.019424 (+ 5199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:02.019427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:02.019449 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:02.019547 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.019553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.038815 (+ 19262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.038819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.040036 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.040053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.041430 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.041434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.041569 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.041595 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:52:02.041669 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:52:02.041755 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5294,"thread_start_us":96,"threads_started":1}
I20260925 22:52:02.042533 18238 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Bootstrap starting.
I20260925 22:52:02.043036 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.009162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.4.219.130:45565)
0925 22:52:02.018931 (+ 9769us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.018938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.018962 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:52:02.019039 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.019045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.019261 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.019419 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:02.019694 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.038642 (+ 18948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.040217 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.041105 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.041212 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.041367 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.042781 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.042889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9657,"thread_start_us":82,"threads_started":1}
I20260925 22:52:02.045605 18238 tablet_bootstrap.cc:654] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:02.046196 17997 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e (DEFAULT_TABLE table=test-table [id=af29fe6bc40c4657b13c709424a90457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:02.047011 18238 log.cc:824] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:02.047379 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbdfcf9efa45e69823b8e021046d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:02.050359 18238 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: No bootstrap required, opened a new log
I20260925 22:52:02.050570 18238 ts_tablet_manager.cc:1397] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:52:02.051807 18239 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Bootstrap starting.
I20260925 22:52:02.054188 18239 tablet_bootstrap.cc:654] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:02.054759 18238 raft_consensus.cc:348] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.055004 18238 raft_consensus.cc:374] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:02.055053 18238 raft_consensus.cc:729] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1f7ad1bb71488eace00748e715d9df, State: Initialized, Role: FOLLOWER
I20260925 22:52:02.055019 18239 log.cc:824] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:02.055512 18238 consensus_queue.cc:260] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.057101 18239 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: No bootstrap required, opened a new log
I20260925 22:52:02.057200 18238 ts_tablet_manager.cc:1428] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:52:02.057289 18239 ts_tablet_manager.cc:1397] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:52:02.057644 18203 heartbeater.cc:503] Master 127.4.219.190:41107 was elected leader, sending a full tablet report...
I20260925 22:52:02.063414 18239 raft_consensus.cc:348] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.063689 18239 raft_consensus.cc:374] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:02.063745 18239 raft_consensus.cc:729] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8583aa29df467c85f6341d3101de2d, State: Initialized, Role: FOLLOWER
I20260925 22:52:02.064250 18239 consensus_queue.cc:260] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.064881 18243 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Bootstrap starting.
I20260925 22:52:02.065351 18239 ts_tablet_manager.cc:1428] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260925 22:52:02.065737 17935 heartbeater.cc:503] Master 127.4.219.190:41107 was elected leader, sending a full tablet report...
I20260925 22:52:02.067931 18243 tablet_bootstrap.cc:654] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:02.069132 18243 log.cc:824] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:02.071422 18243 tablet_bootstrap.cc:492] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: No bootstrap required, opened a new log
I20260925 22:52:02.071633 18243 ts_tablet_manager.cc:1397] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:52:02.076179 18243 raft_consensus.cc:348] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.076524 18243 raft_consensus.cc:374] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:02.076645 18243 raft_consensus.cc:729] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49f8ac6e6ab47d2806148c36fa647ef, State: Initialized, Role: FOLLOWER
I20260925 22:52:02.077196 18243 consensus_queue.cc:260] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.078269 18243 ts_tablet_manager.cc:1428] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:52:02.078644 18069 heartbeater.cc:503] Master 127.4.219.190:41107 was elected leader, sending a full tablet report...
W20260925 22:52:02.149075 17936 tablet.cc:2369] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:52:02.183745 18070 tablet.cc:2369] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:52:02.189188 18204 tablet.cc:2369] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:02.251125 18246 raft_consensus.cc:482] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:02.251367 18246 raft_consensus.cc:504] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.252861 18246 leader_election.cc:288] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1f7ad1bb71488eace00748e715d9df (127.4.219.131:43165), aa8583aa29df467c85f6341d3101de2d (127.4.219.129:41321)
I20260925 22:52:02.258352 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.253155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41321 (local address 127.4.219.130:58919)
0925 22:52:02.253907 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.253921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.254050 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:02.254461 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:02.254465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:02.254485 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:02.254638 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.254645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.255755 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.255761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.257254 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.257264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.257907 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.257910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.258007 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.258029 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:52:02.258123 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:52:02.258190 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":39,"thread_start_us":178,"threads_started":1}
I20260925 22:52:02.258338 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.253742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:58919 (local address 127.4.219.129:41321)
0925 22:52:02.253869 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.253873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.254018 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:52:02.254153 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.254156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.254209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.254298 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:02.254788 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.255606 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.257375 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.257784 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.257923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.257981 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.258137 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.258206 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:52:02.258682 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.253201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:43165 (local address 127.4.219.130:49795)
0925 22:52:02.254240 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.254251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.254372 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:02.254721 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:02.254724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:02.254739 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:02.254851 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.254857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.255882 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.255886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.257254 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:02.257266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.258257 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.258261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:02.258361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.258383 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:52:02.258447 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:52:02.258522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":28,"thread_start_us":105,"threads_started":1}
I20260925 22:52:02.259111 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.253627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49795 (local address 127.4.219.131:43165)
0925 22:52:02.253757 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.253762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.254314 (+ 552us) server_negotiation.cc:407] Connection header received
0925 22:52:02.254484 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.254487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.254545 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.254591 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:02.254971 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.255748 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.257396 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.258022 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.258116 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.258163 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.258892 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.258967 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:52:02.259115 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbdfcf9efa45e69823b8e021046d7e" candidate_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8583aa29df467c85f6341d3101de2d" is_pre_election: true
I20260925 22:52:02.259449 17871 raft_consensus.cc:2454] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49f8ac6e6ab47d2806148c36fa647ef in term 0.
I20260925 22:52:02.259776 18157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbdfcf9efa45e69823b8e021046d7e" candidate_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1f7ad1bb71488eace00748e715d9df" is_pre_election: true
I20260925 22:52:02.260015 17956 leader_election.cc:302] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8583aa29df467c85f6341d3101de2d, c49f8ac6e6ab47d2806148c36fa647ef; no voters:
I20260925 22:52:02.260051 18157 raft_consensus.cc:2454] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c49f8ac6e6ab47d2806148c36fa647ef in term 0.
I20260925 22:52:02.260324 18246 raft_consensus.cc:2781] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:02.260442 18246 raft_consensus.cc:482] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:02.260504 18246 raft_consensus.cc:3037] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:02.261842 18246 raft_consensus.cc:504] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.262238 18246 leader_election.cc:288] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [CANDIDATE]: Term 1 election: Requested vote from peers 2a1f7ad1bb71488eace00748e715d9df (127.4.219.131:43165), aa8583aa29df467c85f6341d3101de2d (127.4.219.129:41321)
I20260925 22:52:02.262573 17871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbdfcf9efa45e69823b8e021046d7e" candidate_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa8583aa29df467c85f6341d3101de2d"
I20260925 22:52:02.262593 18157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbdfcf9efa45e69823b8e021046d7e" candidate_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1f7ad1bb71488eace00748e715d9df"
I20260925 22:52:02.262712 18157 raft_consensus.cc:3037] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:02.262738 17871 raft_consensus.cc:3037] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:02.263845 17871 raft_consensus.cc:2454] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c49f8ac6e6ab47d2806148c36fa647ef in term 1.
I20260925 22:52:02.264000 18157 raft_consensus.cc:2454] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c49f8ac6e6ab47d2806148c36fa647ef in term 1.
I20260925 22:52:02.264163 17956 leader_election.cc:302] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa8583aa29df467c85f6341d3101de2d, c49f8ac6e6ab47d2806148c36fa647ef; no voters:
I20260925 22:52:02.264331 18246 raft_consensus.cc:2781] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:02.264583 18246 raft_consensus.cc:686] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 1 LEADER]: Becoming Leader. State: Replica: c49f8ac6e6ab47d2806148c36fa647ef, State: Running, Role: LEADER
I20260925 22:52:02.264986 18246 consensus_queue.cc:237] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } }
I20260925 22:52:02.268679 17731 catalog_manager.cc:5495] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef reported cstate change: term changed from 0 to 1, leader changed from <none> to c49f8ac6e6ab47d2806148c36fa647ef (127.4.219.130). New cstate: current_term: 1 leader_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c49f8ac6e6ab47d2806148c36fa647ef" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 45565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 } health_report { overall_health: UNKNOWN } } }
I20260925 22:52:02.311129 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:02.307848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.4.219.130:45565)
0925 22:52:02.308006 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:02.308009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:02.308022 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:52:02.308097 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:02.308099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:02.308152 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:02.308241 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:02.308441 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.308975 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.309787 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:02.310021 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:02.310103 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:02.310389 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:02.310806 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:02.310922 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:02.310953 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 22:52:02.311003 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:52:02.317051 17871 raft_consensus.cc:1261] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d [term 1 FOLLOWER]: Refusing update from remote peer c49f8ac6e6ab47d2806148c36fa647ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:02.317054 18157 raft_consensus.cc:1261] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 1 FOLLOWER]: Refusing update from remote peer c49f8ac6e6ab47d2806148c36fa647ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:02.317759 18246 consensus_queue.cc:1035] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8583aa29df467c85f6341d3101de2d" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:02.318005 18256 consensus_queue.cc:1035] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1f7ad1bb71488eace00748e715d9df" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:02.326154 18261 mvcc.cc:205] Tried to move back new op lower bound from 7333383054603399168 to 7333383054400536576. Current Snapshot: MvccSnapshot[applied={T|T < 7333383054603399168}]
I20260925 22:52:02.334573 17737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:02.334780 17737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:02.338418 17737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b160f9b6274fa3a56337a11c99d9fb: Sending DeleteTablet for 3 replicas of tablet 82bbdfcf9efa45e69823b8e021046d7e
I20260925 22:52:02.339517 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:02 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:56420
I20260925 22:52:02.339578 18137 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:02 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:42092
I20260925 22:52:02.340197 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17775
I20260925 22:52:02.340310 18268 tablet_replica.cc:330] T 82bbdfcf9efa45e69823b8e021046d7e P aa8583aa29df467c85f6341d3101de2d: stopping tablet replica
I20260925 22:52:02.341148 17997 tablet_service.cc:1514] Processing DeleteTablet for tablet 82bbdfcf9efa45e69823b8e021046d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:02 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:56256
I20260925 22:52:02.344228 18270 tablet_replica.cc:330] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: stopping tablet replica
I20260925 22:52:02.344614 18270 raft_consensus.cc:2229] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:52:02.345077 18270 raft_consensus.cc:2258] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:02.346698 18270 ts_tablet_manager.cc:1905] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:02.350699 18269 tablet_replica.cc:330] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: stopping tablet replica
I20260925 22:52:02.350260 18270 ts_tablet_manager.cc:1918] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:02.351528 18270 log.cc:1196] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/82bbdfcf9efa45e69823b8e021046d7e
I20260925 22:52:02.351766 18269 raft_consensus.cc:2229] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:02.352028 18270 ts_tablet_manager.cc:1939] T 82bbdfcf9efa45e69823b8e021046d7e P c49f8ac6e6ab47d2806148c36fa647ef: Deleting consensus metadata
I20260925 22:52:02.352133 18269 raft_consensus.cc:2258] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:52:02.352973 17721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:41321 (error 108)
I20260925 22:52:02.353480 17721 catalog_manager.cc:4841] TS c49f8ac6e6ab47d2806148c36fa647ef (127.4.219.130:45565): tablet 82bbdfcf9efa45e69823b8e021046d7e (table test-table [id=af29fe6bc40c4657b13c709424a90457]) successfully deleted
I20260925 22:52:02.353956 18269 ts_tablet_manager.cc:1905] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:52:02.354106 18233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:52:02.353766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:41321 (local address 127.0.0.1:56436)
0925 22:52:02.353901 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:02.353918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:02.353992 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:41321: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:52:02.354238 17721 catalog_manager.cc:4584] TS aa8583aa29df467c85f6341d3101de2d (127.4.219.129:41321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82bbdfcf9efa45e69823b8e021046d7e: Network error: Client connection negotiation failed: client connection to 127.4.219.129:41321: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 22:52:02.354321 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17939
I20260925 22:52:02.359404 18269 ts_tablet_manager.cc:1918] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:02.359565 18269 log.cc:1196] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/82bbdfcf9efa45e69823b8e021046d7e
I20260925 22:52:02.359912 18269 ts_tablet_manager.cc:1939] T 82bbdfcf9efa45e69823b8e021046d7e P 2a1f7ad1bb71488eace00748e715d9df: Deleting consensus metadata
I20260925 22:52:02.361595 17723 catalog_manager.cc:4841] TS 2a1f7ad1bb71488eace00748e715d9df (127.4.219.131:43165): tablet 82bbdfcf9efa45e69823b8e021046d7e (table test-table [id=af29fe6bc40c4657b13c709424a90457]) successfully deleted
I20260925 22:52:02.362280 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18072
I20260925 22:52:02.370098 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 17704
I20260925 22:52:02.378944 4974 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:52:02.723713 4974 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:52:02.723824 4974 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:52:03.068714 4974 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:52:03.068 UTC [16771] LOG: received smart shutdown request
2026-09-25 22:52:03.072 UTC [16771] LOG: background worker "logical replication launcher" (PID 16780) exited with exit code 1
2026-09-25 22:52:03.073 UTC [16775] LOG: shutting down
2026-09-25 22:52:03.235 UTC [16771] LOG: database system is shut down
2026-09-25T22:52:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47694 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:52:03 dist-test-slave-kb1g krb5kdc[18280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:52:03 dist-test-slave-kb1g krb5kdc[18280](info): set up 2 sockets
Sep 25 22:52:03 dist-test-slave-kb1g krb5kdc[18280](info): commencing operation
krb5kdc: starting...
W20260925 22:52:04.392688 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:52:04.515437 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:32857
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:04.623615 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:04.623931 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:04.624004 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:04.627995 18293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:04.628094 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:04.628130 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:04.628170 18293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:04.628201 18293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:04.633478 18293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:04.634645 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:04.635782 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:04.644249 18298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:04.644344 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:04.644372 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:04.644695 18293 server_base.cc:1034] running on GCE node
I20260925 22:52:04.645291 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:04.646325 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:04.647500 18293 hybrid_clock.cc:648] HybridClock initialized: now 1790376724647483 us; error 32 us; skew 500 ppm
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:04.650254 18293 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:04.651262 18293 webserver.cc:460] Webserver started at http://127.4.219.190:33271/ using document root <none> and password file <none>
I20260925 22:52:04.651814 18293 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:04.651877 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:04.652122 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:04.654215 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "0a3e20b3d86e4bc1874267acf62902f7"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "82e49e5ce06967d9b4abe1d8c8c543a5"
server_key_iv: "ff15ef39187182a83319ee295499a5bc"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.654791 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "0a3e20b3d86e4bc1874267acf62902f7"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "82e49e5ce06967d9b4abe1d8c8c543a5"
server_key_iv: "ff15ef39187182a83319ee295499a5bc"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.658504 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:52:04.661090 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:04.662333 18293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:04.662494 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "0a3e20b3d86e4bc1874267acf62902f7"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "82e49e5ce06967d9b4abe1d8c8c543a5"
server_key_iv: "ff15ef39187182a83319ee295499a5bc"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.662611 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:04.672143 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:04.675037 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:04.675189 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:04.681881 18293 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:32857
I20260925 22:52:04.681964 18360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:32857 every 8 connection(s)
I20260925 22:52:04.682883 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:04.685570 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:04.689482 18361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:04.691236 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18293
I20260925 22:52:04.691407 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:52:04.691691 4974 external_mini_cluster.cc:1428] Setting key a8ceb476ca434df39e81cbf2e2ef698f
WARNING: no policy specified for kudu/127.4.219.189@KRBTEST.COM; defaulting to no policy
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.189@KRBTEST.COM" created.
I20260925 22:52:04.705588 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.691954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43984 (local address 127.4.219.190:32857)
0925 22:52:04.692568 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:04.692586 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:04.692681 (+ 95us) server_negotiation.cc:407] Connection header received
0925 22:52:04.693725 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:04.693733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:04.693993 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:04.694202 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:04.695005 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.696504 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.697513 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.698159 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.698330 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.700902 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:04.700907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:04.700918 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:04.700939 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:04.702454 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.703003 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.703011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.703013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.703124 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.703461 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.703464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.703466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.703846 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:04.703997 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:04.704427 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:52:04.704748 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":451,"spinlock_wait_cycles":5504,"thread_start_us":165,"threads_started":1}
I20260925 22:52:04.705592 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.691953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:32857 (local address 127.0.0.1:43984)
0925 22:52:04.692478 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.692563 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:04.693405 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:04.694210 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:04.694214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:04.694545 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:04.694849 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.694863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.696658 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.696661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.697388 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.697396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.698316 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.698319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.698426 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.699061 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:04.699076 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:04.700758 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:04.702559 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.702567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.702576 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.702887 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.703227 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.703230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.703232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.703357 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.703966 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:04.703972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:04.704113 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 22:52:04.704306 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 22:52:04.704859 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":165,"thread_start_us":166,"threads_started":1}
I20260925 22:52:04.708117 18361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:04.709030 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.708676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:59962)
0925 22:52:04.708826 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.708925 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:52:04.709270 18311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
I20260925 22:52:04.710183 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.709913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:59978)
0925 22:52:04.710050 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.710114 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:52:04.710331 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:42255: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
I20260925 22:52:04.720377 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:42255
--webserver_interface=127.4.219.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:04.759605 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } attempt: 1
I20260925 22:52:04.760643 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.760236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:59984)
0925 22:52:04.760434 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.760519 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 22:52:04.761426 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.761172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:59986)
0925 22:52:04.761286 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.761344 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:52:04.761560 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:42255: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
I20260925 22:52:04.829737 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } attempt: 2
I20260925 22:52:04.830600 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.830256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:59994)
0925 22:52:04.830439 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.830515 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 22:52:04.831414 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.831142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:60006)
0925 22:52:04.831268 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.831318 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 22:52:04.831588 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:42255: Network error: Client connection negotiation failed: client connection to 127.4.219.189:42255: connect: Connection refused (error 111)
W20260925 22:52:04.832078 18368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:04.832329 18368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:04.832371 18368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:04.836628 18368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:04.836688 18368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:04.836709 18368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:04.836725 18368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:04.836771 18368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:04.842458 18368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.219.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:04.843523 18368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:04.844591 18368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:04.853212 18374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:04.853227 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:04.853400 18368 server_base.cc:1034] running on GCE node
W20260925 22:52:04.853214 18376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:04.854080 18368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:04.855183 18368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:04.856389 18368 hybrid_clock.cc:648] HybridClock initialized: now 1790376724856356 us; error 53 us; skew 500 ppm
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:04.859148 18368 init.cc:377] Logged in from keytab as kudu/127.4.219.189@KRBTEST.COM (short username kudu)
I20260925 22:52:04.860083 18368 webserver.cc:460] Webserver started at http://127.4.219.189:39685/ using document root <none> and password file <none>
I20260925 22:52:04.860688 18368 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:04.860760 18368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:04.860932 18368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:04.862907 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data/instance:
uuid: "64e386068fef4d8c88d7be6518b6467b"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "8c13445e75ab76a9b9e1eb89ee07086c"
server_key_iv: "27cfc1600f2c7272d9f94be80288cf54"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.863404 18368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal/instance:
uuid: "64e386068fef4d8c88d7be6518b6467b"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "8c13445e75ab76a9b9e1eb89ee07086c"
server_key_iv: "27cfc1600f2c7272d9f94be80288cf54"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.867727 18368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:52:04.870105 18383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:04.871291 18368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:52:04.871388 18368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "64e386068fef4d8c88d7be6518b6467b"
format_stamp: "Formatted at 2026-09-25 22:52:04 on dist-test-slave-kb1g"
server_key: "8c13445e75ab76a9b9e1eb89ee07086c"
server_key_iv: "27cfc1600f2c7272d9f94be80288cf54"
server_key_version: "encryptionkey@0"
I20260925 22:52:04.871472 18368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:04.883328 18368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:04.886228 18368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:04.886372 18368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:04.893381 18368 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.189:42255
I20260925 22:52:04.893405 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.189:42255 every 8 connection(s)
I20260925 22:52:04.894331 18368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
I20260925 22:52:04.897115 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18368
I20260925 22:52:04.897261 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-1/wal/instance
I20260925 22:52:04.897274 18436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:04.897571 4974 external_mini_cluster.cc:1428] Setting key a6396e745f815c8393cbc1a3c42d2246
I20260925 22:52:04.901618 18436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.219.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.188@KRBTEST.COM" created.
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:04.915611 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.903536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.4.219.190:32857)
0925 22:52:04.903667 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:04.903670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:04.904724 (+ 1054us) server_negotiation.cc:407] Connection header received
0925 22:52:04.905678 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:04.905681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:04.905747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:04.905839 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:04.907081 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.907664 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.909001 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.909241 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.909358 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.912022 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:04.912025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:04.912028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:04.912052 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:04.913540 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.914037 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.914042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.914044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.914125 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.914472 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.914476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.914478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.914663 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:04.914769 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:04.915293 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 22:52:04.915414 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:52:04.916461 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.904039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:32857 (local address 127.0.0.1:43986)
0925 22:52:04.904572 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.904626 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:04.905424 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:04.906018 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:04.906025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:04.906342 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:04.906937 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.906952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.907796 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.907799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.908839 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.908849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.909398 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.909402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.909582 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.910208 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:04.910225 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:04.911833 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:04.913656 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.913662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.913677 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.913933 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.914215 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.914220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.914222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.914367 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.914768 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:04.914776 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:04.915040 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 22:52:04.915236 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:52:04.915566 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260925 22:52:04.917949 18436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:04.922758 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:52:04.924064 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:40263
--webserver_interface=127.4.219.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:52:04 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:52:04.929831 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.918516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:60012)
0925 22:52:04.918654 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.918667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:04.918735 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:04.919402 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:04.919406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:04.919561 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:04.919647 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.919652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.920984 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.920987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.921631 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.921640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.922315 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.922317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.922387 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.922884 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:04.922903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:04.924444 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:04.928182 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.928186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.928190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.928385 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.928740 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.928742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.928744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.928823 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.929290 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:04.929293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:04.929379 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 22:52:04.929491 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:52:04.929637 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:52:04.929970 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.918643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.4.219.189:42255)
0925 22:52:04.919027 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:04.919035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:04.919053 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:52:04.919087 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:04.919091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:04.919238 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:04.919389 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:04.919779 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.920690 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.921739 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.922214 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.922299 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.924622 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:04.924633 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:04.924652 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:04.924680 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:04.928060 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.928514 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.928519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.928521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.928647 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.928911 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.928917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.928919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.929164 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:04.929331 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:04.929602 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 22:52:04.929732 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":153,"threads_started":1}
I20260925 22:52:04.932276 18436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:04.933010 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.932759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36364)
0925 22:52:04.932882 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.932956 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:52:04.933243 18385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
I20260925 22:52:04.933499 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.923206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:60018)
0925 22:52:04.923327 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.923343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:04.923465 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:04.923912 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:04.923915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:04.924075 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:04.924716 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.924729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.925771 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.925775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.926427 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:04.926441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.927060 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.927064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:04.927180 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.927644 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:04.927658 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:04.929269 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:04.931664 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.931667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.931669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.931919 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.932246 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:04.932249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:04.932251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:04.932344 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:04.932876 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:04.932881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:04.933004 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:52:04.933100 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:52:04.933264 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:52:04.934304 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.934100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36368)
0925 22:52:04.934200 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.934244 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 22:52:04.934321 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.923436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60018 (local address 127.4.219.189:42255)
0925 22:52:04.923710 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:04.923716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:04.923726 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:52:04.923753 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:04.923755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:04.923786 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:04.923887 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:04.924852 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.925657 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.926546 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:04.926934 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:04.927058 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:04.929436 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:04.929441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:04.929446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:04.929475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:04.931542 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.932030 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.932035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.932038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.932131 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:04.932451 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:04.932457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:04.932461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:04.932740 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:04.933839 (+ 1099us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:04.933926 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 22:52:04.934077 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
W20260925 22:52:04.934623 18436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
I20260925 22:52:04.935185 18361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:04.935940 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.935631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36380)
0925 22:52:04.935779 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.935849 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:52:04.936652 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.936379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36394)
0925 22:52:04.936519 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.936598 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 22:52:04.936812 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
I20260925 22:52:04.967968 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 1
I20260925 22:52:04.968962 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.968509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36396)
0925 22:52:04.968778 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.968859 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260925 22:52:04.969743 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.969503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36410)
0925 22:52:04.969624 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.969681 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:52:04.969882 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
I20260925 22:52:04.983875 18436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 1
I20260925 22:52:04.984809 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.984465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36422)
0925 22:52:04.984665 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.984732 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 22:52:04.985626 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:04.985407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36432)
0925 22:52:04.985515 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:04.985563 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:52:04.985776 18436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
W20260925 22:52:05.042583 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:05.042829 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:05.042868 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:52:05.045030 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 2
I20260925 22:52:05.045882 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.045556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36442)
0925 22:52:05.045719 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.045788 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:52:05.046610 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.046389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36446)
0925 22:52:05.046492 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.046549 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:52:05.046738 18361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
W20260925 22:52:05.047464 18443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:05.047509 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:05.047531 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:05.047549 18443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:05.047564 18443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:05.053011 18443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:32857,127.4.219.189:42255,127.4.219.188:40263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:40263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.219.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:05.053951 18436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 2
I20260925 22:52:05.054003 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:05.054661 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.054410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36452)
0925 22:52:05.054538 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.054598 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:52:05.055089 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:52:05.055374 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.055188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36468)
0925 22:52:05.055274 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.055320 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 22:52:05.055485 18436 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:40263: Network error: Client connection negotiation failed: client connection to 127.4.219.188:40263: connect: Connection refused (error 111)
W20260925 22:52:05.063575 18450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:05.063570 18449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:05.063884 18443 server_base.cc:1034] running on GCE node
W20260925 22:52:05.063570 18452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:05.064523 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:05.065688 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:05.066862 18443 hybrid_clock.cc:648] HybridClock initialized: now 1790376725066848 us; error 29 us; skew 500 ppm
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376725, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:05.069623 18443 init.cc:377] Logged in from keytab as kudu/127.4.219.188@KRBTEST.COM (short username kudu)
I20260925 22:52:05.070551 18443 webserver.cc:460] Webserver started at http://127.4.219.188:40021/ using document root <none> and password file <none>
I20260925 22:52:05.071061 18443 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:05.071116 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:05.071276 18443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:05.073311 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data/instance:
uuid: "e48ef108794447e08c567fe7448946ec"
format_stamp: "Formatted at 2026-09-25 22:52:05 on dist-test-slave-kb1g"
server_key: "077e71339dec11338554b14fe1a5c7e6"
server_key_iv: "204dff7d3f07f6be32badb67cc68fd04"
server_key_version: "encryptionkey@0"
I20260925 22:52:05.073814 18443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal/instance:
uuid: "e48ef108794447e08c567fe7448946ec"
format_stamp: "Formatted at 2026-09-25 22:52:05 on dist-test-slave-kb1g"
server_key: "077e71339dec11338554b14fe1a5c7e6"
server_key_iv: "204dff7d3f07f6be32badb67cc68fd04"
server_key_version: "encryptionkey@0"
I20260925 22:52:05.077754 18443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:52:05.080058 18459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:05.081364 18443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:52:05.081491 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "e48ef108794447e08c567fe7448946ec"
format_stamp: "Formatted at 2026-09-25 22:52:05 on dist-test-slave-kb1g"
server_key: "077e71339dec11338554b14fe1a5c7e6"
server_key_iv: "204dff7d3f07f6be32badb67cc68fd04"
server_key_version: "encryptionkey@0"
I20260925 22:52:05.081601 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:05.101279 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:05.104337 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:05.104528 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:05.111696 18443 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.188:40263
I20260925 22:52:05.111721 18511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.188:40263 every 8 connection(s)
I20260925 22:52:05.112649 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
I20260925 22:52:05.115406 18512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:05.119239 18512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:05.121202 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18443
I20260925 22:52:05.121343 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376566921023-4974-0/minicluster-data/master-2/wal/instance
I20260925 22:52:05.121620 4974 external_mini_cluster.cc:1428] Setting key 2d545b19b7c63b19af7e9b65cb8fedcc
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376725, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
I20260925 22:52:05.132918 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.121135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.4.219.190:32857)
0925 22:52:05.121293 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.121300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.122457 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 22:52:05.123627 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.123630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.123674 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.123741 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.124721 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.125226 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.126275 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.126546 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.126622 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.129284 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.129288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.129290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.129310 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.130832 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.131346 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.131348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.131349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.131416 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.131778 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.131782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.131783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.131981 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.132091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.132622 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.132739 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:52:05.133769 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.121662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:32857 (local address 127.0.0.1:43998)
0925 22:52:05.122236 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.122295 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:05.123407 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:05.123818 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:05.123829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:05.124258 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:05.124572 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.124585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.125335 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.125338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.126160 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.126169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.126677 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.126680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.126870 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.127675 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:05.127690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:05.129138 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:05.130937 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.130945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.130955 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.131253 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.131506 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.131510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.131512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.131681 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.132091 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:05.132099 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:05.132341 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 22:52:05.132587 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 22:52:05.132887 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"spinlock_wait_cycles":6016,"thread_start_us":130,"threads_started":1}
I20260925 22:52:05.133775 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.122319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.4.219.188:40263)
0925 22:52:05.122689 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.122699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.122800 (+ 101us) server_negotiation.cc:407] Connection header received
0925 22:52:05.123407 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.123430 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.123661 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.123884 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.124580 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.125579 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.126400 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.126818 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.126999 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.129682 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.129691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.129706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.129733 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.131620 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.132005 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.132011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.132012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.132093 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.132351 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.132356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.132358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.132674 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.132870 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.133316 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.133446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":285,"thread_start_us":133,"threads_started":1}
I20260925 22:52:05.135025 18512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376725, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:05.144490 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.135552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:42255 (local address 127.0.0.1:60026)
0925 22:52:05.135637 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.135652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:05.135742 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:05.136047 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:05.136050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:05.136234 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:05.136343 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.136351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.137401 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.137405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.138249 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.138258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.138824 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.138826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.138920 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.139483 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:05.139498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:05.141182 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:05.143043 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.143045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.143048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.143232 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.143512 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.143516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.143518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.143596 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.143980 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:05.143983 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:05.144076 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:52:05.144182 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:52:05.144309 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 22:52:05.144495 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.135690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60026 (local address 127.4.219.189:42255)
0925 22:52:05.135843 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.135847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.135860 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:52:05.135906 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.135908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.135947 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.136036 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.136482 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.137296 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.138359 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.138714 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.138816 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.141335 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.141338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.141340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.141395 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.142915 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.143331 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.143335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.143337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.143416 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.143686 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.143691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.143692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.143867 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.143989 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.144234 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.144361 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:52:05.145563 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.136286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.4.219.190:32857)
0925 22:52:05.136423 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.136429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.136443 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:52:05.136482 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.136484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.136573 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.136645 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.137309 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.138129 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.139466 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.139809 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.139927 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.142408 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.142415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.142419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.142449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.143884 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.144303 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.144308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.144309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.144400 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.144720 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.144723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.144725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.144894 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.145102 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.145273 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.145375 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:52:05.145812 18512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:52:05.145879 18361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 3
I20260925 22:52:05.146633 18436 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } attempt: 3
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376725, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
Sep 25 22:52:05 dist-test-slave-kb1g krb5kdc[18280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376724, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
I20260925 22:52:05.156841 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.146314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36506)
0925 22:52:05.146911 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.146924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:05.146996 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:05.147913 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:05.147919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:05.148072 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:05.148142 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.148148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149582 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.149585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.150403 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.150413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.151038 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.151041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.151121 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.151583 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:05.151598 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:05.153052 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:05.155083 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.155093 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.155410 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.155713 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.155720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.155819 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.156323 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:05.156326 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:05.156413 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 22:52:05.156509 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:52:05.156658 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":25}
I20260925 22:52:05.157037 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.146199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36492)
0925 22:52:05.146326 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.146341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:05.146409 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:05.147127 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:05.147129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:05.147257 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:05.147326 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.147331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.148357 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.148360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.149190 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.149199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149848 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.149851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.149945 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.150400 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:05.150414 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:05.151843 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:05.153828 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.153832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.153836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.154761 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.155976 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.155981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.156062 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.156471 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:05.156477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:05.156614 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 22:52:05.156721 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:52:05.156844 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:52:05.157552 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.146488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.4.219.188:40263)
0925 22:52:05.146613 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.146618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.146632 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:52:05.146664 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.146669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.146718 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.146784 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.147434 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.148224 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.149351 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149723 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150157 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.152013 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.152017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.152022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.152050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.153721 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155530 (+ 1809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.155533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.155534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.155868 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.156172 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.156175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.156178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.156365 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.157211 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.157300 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.157404 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":38}
I20260925 22:52:05.158224 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.146691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.4.219.188:40263)
0925 22:52:05.147607 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.147617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.147632 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:52:05.147666 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.147669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.147718 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.147805 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.148254 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149486 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150540 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.150864 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.153181 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.153185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.153188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.153210 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.154975 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155512 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.155518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.155519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.155614 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.155920 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.155926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.155929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.156212 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.156293 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.157859 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.157991 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":855,"spinlock_wait_cycles":2304,"thread_start_us":116,"threads_started":1}
I20260925 22:52:05.158223 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.147086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:40263 (local address 127.0.0.1:36516)
0925 22:52:05.147182 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:05.147197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:05.147264 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:05.147870 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:05.147873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:05.148054 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:05.148143 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.148149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149446 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.149450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.150633 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:05.150643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.151194 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.151199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:05.151785 (+ 586us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.152248 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:05.152507 (+ 259us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:05.154525 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:05.156686 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.156689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.156691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.156927 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.157230 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:05.157233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:05.157235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:05.157317 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:05.157742 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:05.157745 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:05.157833 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:52:05.157953 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:52:05.158076 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":50,"spinlock_wait_cycles":2432}
I20260925 22:52:05.158809 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.147293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.4.219.188:40263)
0925 22:52:05.147617 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.147624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.147634 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:52:05.147666 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.147669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.147718 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.148248 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.148422 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149321 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150742 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.151070 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.153391 (+ 2321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.154650 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.154653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.154656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.154683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.156561 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.157044 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.157049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.157054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.157132 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.157445 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.157451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.157453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.157659 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.157743 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.158534 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.158642 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":6912,"thread_start_us":124,"threads_started":1}
I20260925 22:52:05.160902 18361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: Bootstrap starting.
I20260925 22:52:05.161422 18512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: Bootstrap starting.
I20260925 22:52:05.163726 18361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:05.163726 18512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:05.164618 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.147034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60032 (local address 127.4.219.189:42255)
0925 22:52:05.147515 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.147519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.147530 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:52:05.147560 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.147563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.147599 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.147665 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.148400 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.149183 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150217 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.150859 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.150946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.154546 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.154550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.154552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.154576 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.156193 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.156724 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.156731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.156735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.156802 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.157115 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.157118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.157119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.157324 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.164226 (+ 6902us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.164312 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.164431 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":409}
I20260925 22:52:05.164662 18361 log.cc:824] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:05.164662 18512 log.cc:824] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:05.166687 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: Bootstrap starting.
I20260925 22:52:05.167282 18361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: No bootstrap required, opened a new log
I20260925 22:52:05.167282 18512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: No bootstrap required, opened a new log
I20260925 22:52:05.168044 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:52:05.169375 18436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:05.170228 18436 log.cc:824] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:05.171692 18361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.171953 18361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:05.172013 18361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3e20b3d86e4bc1874267acf62902f7, State: Initialized, Role: FOLLOWER
I20260925 22:52:05.172058 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: No bootstrap required, opened a new log
I20260925 22:52:05.172227 18512 raft_consensus.cc:348] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.172499 18512 raft_consensus.cc:374] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:05.172587 18512 raft_consensus.cc:729] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48ef108794447e08c567fe7448946ec, State: Initialized, Role: FOLLOWER
I20260925 22:52:05.172653 18361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.173043 18512 consensus_queue.cc:260] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.173259 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.173451 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.174018 18361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:05.174091 18512 sys_catalog.cc:564] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:05.177055 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.177179 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.176054 18436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.177444 18436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:05.177508 18436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e386068fef4d8c88d7be6518b6467b, State: Initialized, Role: FOLLOWER
I20260925 22:52:05.177925 18436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
W20260925 22:52:05.176594 18314 tablet.cc:2369] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:05.179152 18436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:52:05.181452 18549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:52:05.182456 18548 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:52:05.182551 18548 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 22:52:05.182636 18549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:52:05.182780 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.182940 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.181748 18443 master_runner.cc:186] Error getting master registration for 127.4.219.189:42255: OK, instance_id { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" instance_seqno: 1790376724891348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:52:05.184495 18443 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 22:52:05.188853 18566 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:52:05.188938 18566 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:52:05.191058 18368 master_runner.cc:263] No leader master found from master 64e386068fef4d8c88d7be6518b6467b
I20260925 22:52:05.191128 18368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:52:05.182237 18293 master_runner.cc:186] Error getting master registration for 127.4.219.189:42255: OK, instance_id { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" instance_seqno: 1790376724891348 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:52:05.191437 18293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:52:05.195142 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.176672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.4.219.190:32857)
0925 22:52:05.176805 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.176811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.181974 (+ 5163us) server_negotiation.cc:407] Connection header received
0925 22:52:05.185327 (+ 3353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.185330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.185369 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.185451 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.186505 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.187207 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.188473 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.188827 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.188942 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.190803 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.190807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.190812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.190838 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.192602 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.193416 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.193420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.193422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.193509 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.193805 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.193809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.193811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.194041 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.194164 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.194774 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.194908 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:52:05.198700 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.177135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.4.219.189:42255)
0925 22:52:05.177246 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.177251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.185076 (+ 7825us) server_negotiation.cc:407] Connection header received
0925 22:52:05.185116 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.185118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.185161 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.185230 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.186428 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.187152 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.188271 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.188620 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.188694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.194514 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.194519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.194522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.194548 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.196271 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.196810 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.196814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.196816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.196909 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.197215 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.197219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.197220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.197699 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.197823 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.198432 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.198551 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:52:05.201007 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:05.177823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.4.219.188:40263)
0925 22:52:05.177927 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:05.177945 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:05.184789 (+ 6844us) server_negotiation.cc:407] Connection header received
0925 22:52:05.184834 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:05.184839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:05.184911 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:05.184989 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:05.189932 (+ 4943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.190656 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.191829 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:05.192190 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:05.192290 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:05.197183 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:05.197188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:05.197191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:05.197217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:05.198863 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.199504 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.199508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.199510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.199614 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:05.199918 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:05.199921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:05.199923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:05.200168 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:05.200279 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:05.200683 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 22:52:05.200841 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:52:05.219339 18527 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:05.219533 18527 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.220642 18527 leader_election.cc:288] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a3e20b3d86e4bc1874267acf62902f7 (127.4.219.190:32857), 64e386068fef4d8c88d7be6518b6467b (127.4.219.189:42255)
I20260925 22:52:05.221310 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e48ef108794447e08c567fe7448946ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e386068fef4d8c88d7be6518b6467b" is_pre_election: true
I20260925 22:52:05.221618 18410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e48ef108794447e08c567fe7448946ec in term 0.
I20260925 22:52:05.222055 18462 leader_election.cc:302] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64e386068fef4d8c88d7be6518b6467b, e48ef108794447e08c567fe7448946ec; no voters:
I20260925 22:52:05.223014 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e48ef108794447e08c567fe7448946ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3e20b3d86e4bc1874267acf62902f7" is_pre_election: true
I20260925 22:52:05.223330 18335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e48ef108794447e08c567fe7448946ec in term 0.
I20260925 22:52:05.224026 18527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:05.224113 18527 raft_consensus.cc:482] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:05.224155 18527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:05.225580 18527 raft_consensus.cc:504] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.225920 18527 leader_election.cc:288] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [CANDIDATE]: Term 1 election: Requested vote from peers 0a3e20b3d86e4bc1874267acf62902f7 (127.4.219.190:32857), 64e386068fef4d8c88d7be6518b6467b (127.4.219.189:42255)
I20260925 22:52:05.226450 18410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e48ef108794447e08c567fe7448946ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64e386068fef4d8c88d7be6518b6467b"
I20260925 22:52:05.226611 18410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:05.227982 18410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e48ef108794447e08c567fe7448946ec in term 1.
I20260925 22:52:05.228356 18462 leader_election.cc:302] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64e386068fef4d8c88d7be6518b6467b, e48ef108794447e08c567fe7448946ec; no voters:
I20260925 22:52:05.228888 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e48ef108794447e08c567fe7448946ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3e20b3d86e4bc1874267acf62902f7"
I20260925 22:52:05.229027 18335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:05.230326 18335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e48ef108794447e08c567fe7448946ec in term 1.
I20260925 22:52:05.233117 18527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:05.233393 18527 raft_consensus.cc:686] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [term 1 LEADER]: Becoming Leader. State: Replica: e48ef108794447e08c567fe7448946ec, State: Running, Role: LEADER
I20260925 22:52:05.233795 18527 consensus_queue.cc:237] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } }
I20260925 22:52:05.235620 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48ef108794447e08c567fe7448946ec. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.235807 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:05.236750 18570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:05.241227 18570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:05.247524 18410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [term 1 FOLLOWER]: Refusing update from remote peer e48ef108794447e08c567fe7448946ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:05.248267 18568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:05.254993 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48ef108794447e08c567fe7448946ec. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.255128 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.256207 18335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [term 1 FOLLOWER]: Refusing update from remote peer e48ef108794447e08c567fe7448946ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:05.256822 18568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:05.260615 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e48ef108794447e08c567fe7448946ec. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.260746 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.266301 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.266417 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:05.269709 18526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.269812 18526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.270604 18571 mvcc.cc:205] Tried to move back new op lower bound from 7333383066606096384 to 7333383066561118208. Current Snapshot: MvccSnapshot[applied={T|T < 7333383066606096384}]
I20260925 22:52:05.272753 18574 mvcc.cc:205] Tried to move back new op lower bound from 7333383066606096384 to 7333383066561118208. Current Snapshot: MvccSnapshot[applied={T|T < 7333383066606096384}]
I20260925 22:52:05.273727 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.273833 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:05.274988 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e48ef108794447e08c567fe7448946ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a3e20b3d86e4bc1874267acf62902f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 32857 } } peers { permanent_uuid: "64e386068fef4d8c88d7be6518b6467b" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 42255 } } peers { permanent_uuid: "e48ef108794447e08c567fe7448946ec" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 40263 } } }
I20260925 22:52:05.275118 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:52:05.277330 18572 mvcc.cc:205] Tried to move back new op lower bound from 7333383066606096384 to 7333383066561118208. Current Snapshot: MvccSnapshot[applied={T|T < 7333383066606096384}]
I20260925 22:52:05.282864 18570 catalog_manager.cc:1347] Generated new cluster ID: 68b15455436e46be878644c0dfee5617
I20260925 22:52:05.282928 18570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:05.312657 18570 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:52:05.314090 18570 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:05.334592 18570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e48ef108794447e08c567fe7448946ec: Generated new TSK 0
I20260925 22:52:05.335546 18570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:52:06.183856 18549 catalog_manager.cc:1259] Loaded cluster ID: 68b15455436e46be878644c0dfee5617
I20260925 22:52:06.183970 18549 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: loading cluster ID for follower catalog manager: success
I20260925 22:52:06.185722 18549 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: acquiring CA information for follower catalog manager: success
I20260925 22:52:06.187012 18549 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a3e20b3d86e4bc1874267acf62902f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:52:06.190161 18566 catalog_manager.cc:1259] Loaded cluster ID: 68b15455436e46be878644c0dfee5617
I20260925 22:52:06.190246 18566 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: loading cluster ID for follower catalog manager: success
I20260925 22:52:06.191795 18566 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: acquiring CA information for follower catalog manager: success
I20260925 22:52:06.192937 18566 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64e386068fef4d8c88d7be6518b6467b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:52:06.407186 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18293
I20260925 22:52:06.413480 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18368
I20260925 22:52:06.419651 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18443
2026-09-25T22:52:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3068 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:52:06 dist-test-slave-kb1g krb5kdc[18585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:52:06 dist-test-slave-kb1g krb5kdc[18585](info): set up 2 sockets
Sep 25 22:52:06 dist-test-slave-kb1g krb5kdc[18585](info): commencing operation
krb5kdc: starting...
W20260925 22:52:07.460880 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:52:07.581739 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39603
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:07.690584 18598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:07.690836 18598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:07.690876 18598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:07.694880 18598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:07.694941 18598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:07.694962 18598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:07.694977 18598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:07.694993 18598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:07.700088 18598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:07.701046 18598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:07.701969 18598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:07.710071 18606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:07.710062 18603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:07.710198 18598 server_base.cc:1034] running on GCE node
W20260925 22:52:07.710062 18604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:07.711117 18598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:07.712167 18598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:07.713356 18598 hybrid_clock.cc:648] HybridClock initialized: now 1790376727713314 us; error 53 us; skew 500 ppm
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376727, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:07.715975 18598 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:07.716934 18598 webserver.cc:460] Webserver started at http://127.4.219.190:44485/ using document root <none> and password file <none>
I20260925 22:52:07.717473 18598 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:07.717561 18598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:07.717789 18598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:07.719738 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "ec96fa7d6f93412a954ddd2776b669b3"
format_stamp: "Formatted at 2026-09-25 22:52:07 on dist-test-slave-kb1g"
server_key: "92e85fc76ddfadabec908dbad4eaef3c"
server_key_iv: "7e5b9d7b8ca4ab53dc13e16d5a2f0e8f"
server_key_version: "encryptionkey@0"
I20260925 22:52:07.720319 18598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "ec96fa7d6f93412a954ddd2776b669b3"
format_stamp: "Formatted at 2026-09-25 22:52:07 on dist-test-slave-kb1g"
server_key: "92e85fc76ddfadabec908dbad4eaef3c"
server_key_iv: "7e5b9d7b8ca4ab53dc13e16d5a2f0e8f"
server_key_version: "encryptionkey@0"
I20260925 22:52:07.723948 18598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:52:07.726339 18613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:07.727502 18598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:07.727633 18598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "ec96fa7d6f93412a954ddd2776b669b3"
format_stamp: "Formatted at 2026-09-25 22:52:07 on dist-test-slave-kb1g"
server_key: "92e85fc76ddfadabec908dbad4eaef3c"
server_key_iv: "7e5b9d7b8ca4ab53dc13e16d5a2f0e8f"
server_key_version: "encryptionkey@0"
I20260925 22:52:07.727742 18598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:07.752205 18598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:07.755599 18598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:07.755833 18598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:07.763051 18598 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39603
I20260925 22:52:07.763080 18665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39603 every 8 connection(s)
I20260925 22:52:07.764005 18598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:07.766809 18666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:07.767822 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18598
I20260925 22:52:07.767954 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:52:07.768289 4974 external_mini_cluster.cc:1428] Setting key b8c275ed47f58781c6baa790fec0c516
I20260925 22:52:07.772948 18666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3: Bootstrap starting.
I20260925 22:52:07.775110 18666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:07.775931 18666 log.cc:824] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:07.777994 18666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3: No bootstrap required, opened a new log
I20260925 22:52:07.781809 18666 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } }
I20260925 22:52:07.782034 18666 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:07.782092 18666 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec96fa7d6f93412a954ddd2776b669b3, State: Initialized, Role: FOLLOWER
I20260925 22:52:07.782279 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:07.769702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.4.219.190:39603)
0925 22:52:07.770137 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:07.770152 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:07.770187 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:52:07.770821 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:07.770838 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:07.771175 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:07.771520 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:07.772322 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:07.773406 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:07.774259 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:07.774727 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:07.774889 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:07.777489 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:07.777497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:07.777516 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:07.777542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:07.779257 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:07.779700 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:07.779706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:07.779719 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:07.779829 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:07.780115 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:07.780119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:07.780121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:07.780578 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:07.780800 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:07.781080 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 22:52:07.781365 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260925 22:52:07.782586 18666 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } }
I20260925 22:52:07.782742 18666 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:07.782812 18666 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:07.782907 18666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:07.783913 18666 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } }
I20260925 22:52:07.784308 18666 leader_election.cc:302] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec96fa7d6f93412a954ddd2776b669b3; no voters:
I20260925 22:52:07.784686 18666 leader_election.cc:288] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:52:07.784770 18671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:07.785243 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:52:07.785357 18671 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [term 1 LEADER]: Becoming Leader. State: Replica: ec96fa7d6f93412a954ddd2776b669b3, State: Running, Role: LEADER
I20260925 22:52:07.785648 18671 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } }
I20260925 22:52:07.785946 18666 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:07.787254 18673 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec96fa7d6f93412a954ddd2776b669b3. Latest consensus state: current_term: 1 leader_uuid: "ec96fa7d6f93412a954ddd2776b669b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } } }
I20260925 22:52:07.787386 18673 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:07.787724 18672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec96fa7d6f93412a954ddd2776b669b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec96fa7d6f93412a954ddd2776b669b3" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39603 } } }
I20260925 22:52:07.787777 18681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:07.787832 18672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:07.790889 18681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:07.795857 18681 catalog_manager.cc:1347] Generated new cluster ID: 300688adc9534862930680c5d4ab2658
I20260925 22:52:07.795920 18681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:07.817744 18681 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:52:07.818711 18681 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:07.828668 18681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec96fa7d6f93412a954ddd2776b669b3: Generated new TSK 0
I20260925 22:52:07.829384 18681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:52:07.841312 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18598
2026-09-25T22:52:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18693](info): set up 2 sockets
Sep 25 22:52:07 dist-test-slave-kb1g krb5kdc[18693](info): commencing operation
krb5kdc: starting...
W20260925 22:52:08.884887 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:52:08 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:52:09.005621 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:09.115232 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:09.115512 18706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:09.115553 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:09.119616 18706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:09.119679 18706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:09.119699 18706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:09.119716 18706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:09.119731 18706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:09.125061 18706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:09.126063 18706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:09.127028 18706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:09.135720 18712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.135748 18711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.135898 18706 server_base.cc:1034] running on GCE node
W20260925 22:52:09.135757 18714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.136627 18706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:09.137899 18706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:09.139091 18706 hybrid_clock.cc:648] HybridClock initialized: now 1790376729139041 us; error 62 us; skew 500 ppm
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:09.146313 18706 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:09.147307 18706 webserver.cc:460] Webserver started at http://127.4.219.190:38627/ using document root <none> and password file <none>
I20260925 22:52:09.147907 18706 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:09.147974 18706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:09.148188 18706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:09.150408 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "7eeae6a1d8fb468a997527d46ba4494e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "3204ee93757ad753dffec37122b49ebb"
server_key_iv: "4672436ddccd0f2583bcfb34797af9c1"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.150991 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "7eeae6a1d8fb468a997527d46ba4494e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "3204ee93757ad753dffec37122b49ebb"
server_key_iv: "4672436ddccd0f2583bcfb34797af9c1"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.154807 18706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:52:09.157191 18721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.158510 18706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:09.158641 18706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "7eeae6a1d8fb468a997527d46ba4494e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "3204ee93757ad753dffec37122b49ebb"
server_key_iv: "4672436ddccd0f2583bcfb34797af9c1"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.158746 18706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:09.179692 18706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:09.183097 18706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:09.183312 18706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:09.190728 18706 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39645
I20260925 22:52:09.191072 18773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39645 every 8 connection(s)
I20260925 22:52:09.191810 18706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:09.195019 18774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:09.201254 18774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap starting.
I20260925 22:52:09.202099 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18706
I20260925 22:52:09.202204 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:52:09.202553 4974 external_mini_cluster.cc:1428] Setting key 182ec4b95f50fd79f5d4e95b089eb491
I20260925 22:52:09.204325 18774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:09.205179 18774 log.cc:824] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:09.207214 18774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: No bootstrap required, opened a new log
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:09.211421 18774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:09.211663 18774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:09.211738 18774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Initialized, Role: FOLLOWER
I20260925 22:52:09.212261 18774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:09.212437 18774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:09.212565 18774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:09.212714 18774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:09.213790 18774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:09.214145 18774 leader_election.cc:302] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeae6a1d8fb468a997527d46ba4494e; no voters:
I20260925 22:52:09.214504 18774 leader_election.cc:288] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:52:09.214572 18779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:09.214790 18779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 LEADER]: Becoming Leader. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Running, Role: LEADER
I20260925 22:52:09.215137 18779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:09.215886 18774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:09.217127 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:09.217270 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:09.217517 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.203763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.4.219.190:39645)
0925 22:52:09.204173 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.204188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.204223 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:52:09.204969 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.204989 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.205351 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.205730 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:09.206538 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.207586 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.208470 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.209001 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.209168 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.211985 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:09.211990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:09.212004 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:09.212031 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:09.213993 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.214528 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.214534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.214540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.214695 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.215067 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.215072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.215074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.215535 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:09.215675 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.216150 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.216487 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":139,"threads_started":1}
I20260925 22:52:09.217583 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeae6a1d8fb468a997527d46ba4494e. Latest consensus state: current_term: 1 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:09.217669 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
W20260925 22:52:09.221035 18795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:52:09.221140 18795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:52:09.221215 18788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:09.221892 18788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:09.227182 18788 catalog_manager.cc:1347] Generated new cluster ID: 1044e499d6324c9db907a65d2c9658ce
I20260925 22:52:09.227267 18788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:09.257095 18788 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:52:09.258078 18788 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:09.270390 18788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Generated new TSK 0
I20260925 22:52:09.271085 18788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:52:09.303325 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:09.413574 18800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:09.413825 18800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:09.413868 18800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:09.418169 18800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:09.418227 18800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:09.418315 18800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:09.423523 18800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:09.424705 18800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:09.425757 18800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:09.435600 18806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.435726 18800 server_base.cc:1034] running on GCE node
W20260925 22:52:09.435546 18808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.435564 18805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.436316 18800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:09.436996 18800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:09.438179 18800 hybrid_clock.cc:648] HybridClock initialized: now 1790376729438151 us; error 45 us; skew 500 ppm
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:09.441129 18800 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:09.442029 18800 webserver.cc:460] Webserver started at http://127.4.219.129:37627/ using document root <none> and password file <none>
I20260925 22:52:09.442603 18800 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:09.442691 18800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:09.442922 18800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:09.444993 18800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "f03679e2b39041099ebcf6bf8b05d180"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9f30fc770c3a439f6591b83800d41ce2"
server_key_iv: "c3372c2a9532e1de466ed1b3a06d0afd"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.445538 18800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "f03679e2b39041099ebcf6bf8b05d180"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9f30fc770c3a439f6591b83800d41ce2"
server_key_iv: "c3372c2a9532e1de466ed1b3a06d0afd"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.449237 18800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:09.451650 18815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.452903 18800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:09.453042 18800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "f03679e2b39041099ebcf6bf8b05d180"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9f30fc770c3a439f6591b83800d41ce2"
server_key_iv: "c3372c2a9532e1de466ed1b3a06d0afd"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.453152 18800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:09.466948 18800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:09.470082 18800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:09.470288 18800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:09.470881 18800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:09.471709 18800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:09.471778 18800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.471849 18800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:09.471902 18800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.481007 18800 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:33529
I20260925 22:52:09.481025 18928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:33529 every 8 connection(s)
I20260925 22:52:09.481936 18800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:09.490077 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18800
I20260925 22:52:09.490227 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:52:09.490588 4974 external_mini_cluster.cc:1428] Setting key b51ad65d261069b54fbb92122afe36c8
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:09.496799 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.483753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:51365 (local address 127.4.219.190:39645)
0925 22:52:09.483889 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.483893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.484760 (+ 867us) server_negotiation.cc:407] Connection header received
0925 22:52:09.486092 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.486095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.486154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.486243 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:09.487430 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.487975 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.489185 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.489433 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.489547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.492481 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:09.492488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:09.492491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:09.492521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:09.494220 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.494884 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.494887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.494889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.494968 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.495399 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.495402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.495403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.495587 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:09.495676 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.496367 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.496500 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:52:09.497884 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.484133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:51365)
0925 22:52:09.484620 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.484664 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.485877 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.486434 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.486441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.486800 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:09.487279 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.487292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.488102 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.488106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.489059 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.489068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.489586 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.489590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.489758 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.490526 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:09.490550 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:09.492291 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:09.494352 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.494360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.494379 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.494756 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.495066 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.495072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.495075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.495275 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.495704 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:09.495713 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:09.496049 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.496263 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.496705 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260925 22:52:09.499292 18929 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:09.499571 18929 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:09.500178 18929 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:52:09.502034 18738 ts_manager.cc:194] Registered new tserver with Master: f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:52:09.503371 18738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:51365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:52:09.519573 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:09.632128 18934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:09.632385 18934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:09.632426 18934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:09.636504 18934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:09.636581 18934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:09.636642 18934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:09.641840 18934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:09.642915 18934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:09.643903 18934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:09.653806 18942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.653815 18939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.653817 18940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.654069 18934 server_base.cc:1034] running on GCE node
I20260925 22:52:09.654556 18934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:09.655212 18934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:09.656395 18934 hybrid_clock.cc:648] HybridClock initialized: now 1790376729656357 us; error 49 us; skew 500 ppm
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:09.659219 18934 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:09.660102 18934 webserver.cc:460] Webserver started at http://127.4.219.130:37133/ using document root <none> and password file <none>
I20260925 22:52:09.660735 18934 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:09.660804 18934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:09.660985 18934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:09.663087 18934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "d7b35fc288894c20ab4f2793a8deb728"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9e0b1615c87602ec52b6f31266b29018"
server_key_iv: "1aab064b24ebfbe760d740448887b3b3"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.663749 18934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "d7b35fc288894c20ab4f2793a8deb728"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9e0b1615c87602ec52b6f31266b29018"
server_key_iv: "1aab064b24ebfbe760d740448887b3b3"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.667574 18934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:52:09.670039 18949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.671308 18934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:52:09.671415 18934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "d7b35fc288894c20ab4f2793a8deb728"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9e0b1615c87602ec52b6f31266b29018"
server_key_iv: "1aab064b24ebfbe760d740448887b3b3"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.671504 18934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:09.683573 18934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:09.686872 18934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:09.687022 18934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:09.687528 18934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:09.688283 18934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:09.688329 18934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.688369 18934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:09.688385 18934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.697791 18934 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40925
I20260925 22:52:09.697909 19062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40925 every 8 connection(s)
I20260925 22:52:09.698746 18934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:09.706408 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 18934
I20260925 22:52:09.706533 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:52:09.706805 4974 external_mini_cluster.cc:1428] Setting key b4213c3fe25c28c6789cd9384c98ba32
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:09.712730 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.700659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49727 (local address 127.4.219.190:39645)
0925 22:52:09.700801 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.700805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.701686 (+ 881us) server_negotiation.cc:407] Connection header received
0925 22:52:09.702937 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.702940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.702985 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.703060 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:09.704160 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.704725 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.705788 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.706010 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.706104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.708714 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:09.708720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:09.708726 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:09.708756 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:09.710325 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.710949 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.710951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.710953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.711022 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.711423 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.711429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.711433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.711691 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:09.711798 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.712322 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.712462 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:52:09.713495 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.701074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:49727)
0925 22:52:09.701531 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.701573 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.702738 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.703189 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.703195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.703558 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:09.704027 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.704040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.704828 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.704830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.705683 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.705692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.706169 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.706175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.706336 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.706989 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:09.707006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:09.708497 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:09.710437 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.710445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.710462 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.710833 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.711112 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.711117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.711119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.711293 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.711822 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:09.711827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:09.712066 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.712230 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.712480 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":118,"threads_started":1}
I20260925 22:52:09.714869 19063 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:09.715214 19063 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:09.715902 19063 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
I20260925 22:52:09.717198 18738 ts_manager.cc:194] Registered new tserver with Master: d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:52:09.717885 18738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:49727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:52:09.735354 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:09.845541 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:09.845800 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:09.845881 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:09.850064 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:09.850135 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:09.850255 19068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:09.855567 19068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:09.856696 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:09.857767 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:52:09.867621 19068 server_base.cc:1034] running on GCE node
W20260925 22:52:09.867594 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.867583 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:09.867580 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:09.868271 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:09.869022 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:09.870188 19068 hybrid_clock.cc:648] HybridClock initialized: now 1790376729870154 us; error 48 us; skew 500 ppm
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:09.873001 19068 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:09.873798 19068 webserver.cc:460] Webserver started at http://127.4.219.131:36927/ using document root <none> and password file <none>
I20260925 22:52:09.874372 19068 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:09.874432 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:09.874588 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:09.876680 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "803bce081f4946018d306cfa88c2751e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "50c82170319ee01c97e068774e71ba77"
server_key_iv: "56af342dfc7503d1173691f60bc0066c"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.877188 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "803bce081f4946018d306cfa88c2751e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "50c82170319ee01c97e068774e71ba77"
server_key_iv: "56af342dfc7503d1173691f60bc0066c"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.880746 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:09.883198 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.884575 19068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:52:09.884735 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "803bce081f4946018d306cfa88c2751e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "50c82170319ee01c97e068774e71ba77"
server_key_iv: "56af342dfc7503d1173691f60bc0066c"
server_key_version: "encryptionkey@0"
I20260925 22:52:09.884851 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:09.903708 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:09.907163 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:09.907375 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:09.907997 19068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:09.908849 19068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:09.908895 19068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.908963 19068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:09.909006 19068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:09.917770 19068 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:45309
I20260925 22:52:09.917819 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:45309 every 8 connection(s)
I20260925 22:52:09.918671 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:09.921665 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19068
I20260925 22:52:09.921765 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:52:09.922049 4974 external_mini_cluster.cc:1428] Setting key 7ae20b5a1bb4ca36bdca425d645b905d
Sep 25 22:52:09 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376729, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:09.932116 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.920389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:41677 (local address 127.4.219.190:39645)
0925 22:52:09.920588 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.920593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.921410 (+ 817us) server_negotiation.cc:407] Connection header received
0925 22:52:09.922913 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.922917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.922974 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.923086 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:09.924266 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.924865 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.926017 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.926244 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.926356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.928796 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:09.928799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:09.928801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:09.928833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:09.930078 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.930606 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.930612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.930613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.930694 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.931038 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.931043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.931045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.931240 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:09.931351 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.931838 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.931955 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 22:52:09.933007 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.920790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:41677)
0925 22:52:09.921246 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.921286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.922703 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.923268 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.923274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.923616 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:09.924119 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.924132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.924987 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.924998 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.925889 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.925898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.926392 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.926396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.926565 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.927104 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:09.927120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:09.928610 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:09.930215 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.930220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.930234 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.930495 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.930790 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:09.930795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:09.930797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:09.930944 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:09.931362 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:09.931370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:09.931603 (+ 233us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.931799 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.932114 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260925 22:52:09.934134 19197 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:09.934404 19197 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:09.935029 19197 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
I20260925 22:52:09.936091 18738 ts_manager.cc:194] Registered new tserver with Master: 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309)
I20260925 22:52:09.936712 18738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:41677
I20260925 22:52:09.948024 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:52:09.958815 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.950558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.4.219.190:39645)
0925 22:52:09.950664 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.950668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.951109 (+ 441us) server_negotiation.cc:407] Connection header received
0925 22:52:09.951255 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.951259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.951319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.951409 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:09.952313 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.953085 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.954225 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.954440 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.954522 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.955866 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:09.955869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:09.955871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:09.955893 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:09.957216 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.957617 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.957619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.957620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.957738 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:09.958027 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:09.958030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:09.958031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:09.958271 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:09.958384 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.958579 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.958684 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:52:09.961791 18738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:52:09.965022 18738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:52:09.979176 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.973530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40925 (local address 127.0.0.1:50848)
0925 22:52:09.974011 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.974057 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.974185 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.974815 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.974821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.974850 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:09.974988 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.974996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.976428 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.976431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.977528 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.977536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.978637 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.978641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.978751 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.978800 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.978890 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.979022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":71,"threads_started":1}
I20260925 22:52:09.979543 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.973677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:45309 (local address 127.0.0.1:40712)
0925 22:52:09.974461 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.974473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.974547 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.975492 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.975494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.975513 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:09.975616 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.975623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.976868 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.976871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.978006 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.978016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.979093 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.979097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.979186 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.979214 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.979312 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.979396 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260925 22:52:09.979756 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.974207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33529 (local address 127.0.0.1:47502)
0925 22:52:09.975069 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:09.975082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:09.975159 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:09.975875 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:09.975878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:09.975895 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:09.975978 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.975983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.977461 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.977465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.978411 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:09.978419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.979382 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.979385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:09.979474 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.979494 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:52:09.979541 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 22:52:09.979614 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260925 22:52:09.980062 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.974439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.4.219.131:45309)
0925 22:52:09.974753 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.974777 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.974801 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:52:09.974880 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.974884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.975161 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.975321 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:09.975741 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.976698 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.978142 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.978967 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.979068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.979208 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.979776 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.979891 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":139,"threads_started":1}
I20260925 22:52:09.980091 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.973685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.4.219.130:40925)
0925 22:52:09.974311 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.974323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.974357 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:52:09.974436 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.974441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.974664 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.974816 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:09.975129 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.976287 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.977656 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.978499 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.979654 (+ 1155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.979810 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.979871 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.979966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":167,"threads_started":1}
I20260925 22:52:09.980479 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:09.974939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.4.219.129:33529)
0925 22:52:09.975457 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:09.975471 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:09.975494 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:52:09.975563 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:09.975568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:09.975724 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:09.976103 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:09.976239 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.977311 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.978541 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:09.979256 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:09.980027 (+ 771us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:09.980187 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:09.980253 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:52:09.980350 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":103,"threads_started":1}
I20260925 22:52:09.983176 18863 tablet_service.cc:1467] Processing CreateTablet for tablet 0342786d717c4b98a057af9d7aac2b23 (DEFAULT_TABLE table=test-table [id=2cf900cc5c08423db7c73c9b62b83146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:09.983134 18997 tablet_service.cc:1467] Processing CreateTablet for tablet 0342786d717c4b98a057af9d7aac2b23 (DEFAULT_TABLE table=test-table [id=2cf900cc5c08423db7c73c9b62b83146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:09.983134 19131 tablet_service.cc:1467] Processing CreateTablet for tablet 0342786d717c4b98a057af9d7aac2b23 (DEFAULT_TABLE table=test-table [id=2cf900cc5c08423db7c73c9b62b83146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:09.984144 18863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0342786d717c4b98a057af9d7aac2b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:09.984165 19131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0342786d717c4b98a057af9d7aac2b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:09.984158 18997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0342786d717c4b98a057af9d7aac2b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:09.991145 19214 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap starting.
I20260925 22:52:09.991858 19215 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Bootstrap starting.
I20260925 22:52:09.992273 19216 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Bootstrap starting.
I20260925 22:52:09.994225 19214 tablet_bootstrap.cc:654] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:09.994400 19215 tablet_bootstrap.cc:654] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:09.994865 19216 tablet_bootstrap.cc:654] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:09.995484 19215 log.cc:824] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:09.995556 19214 log.cc:824] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:09.995625 19216 log.cc:824] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:09.997525 19215 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: No bootstrap required, opened a new log
I20260925 22:52:09.997628 19216 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: No bootstrap required, opened a new log
I20260925 22:52:09.997628 19214 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: No bootstrap required, opened a new log
I20260925 22:52:09.997776 19215 ts_tablet_manager.cc:1397] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:52:09.997812 19214 ts_tablet_manager.cc:1397] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:52:09.997812 19216 ts_tablet_manager.cc:1397] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:52:10.000972 19215 raft_consensus.cc:348] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.001240 19215 raft_consensus.cc:374] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.001312 19215 raft_consensus.cc:729] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803bce081f4946018d306cfa88c2751e, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.001803 19215 consensus_queue.cc:260] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.001847 19216 raft_consensus.cc:348] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.001842 19214 raft_consensus.cc:348] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.002103 19214 raft_consensus.cc:374] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.002103 19216 raft_consensus.cc:374] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.002162 19216 raft_consensus.cc:729] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b35fc288894c20ab4f2793a8deb728, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.002203 19214 raft_consensus.cc:729] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03679e2b39041099ebcf6bf8b05d180, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.002620 19216 consensus_queue.cc:260] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.002620 19214 consensus_queue.cc:260] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.002889 19215 ts_tablet_manager.cc:1428] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:52:10.003242 18929 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
I20260925 22:52:10.003504 19214 ts_tablet_manager.cc:1428] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:52:10.003504 19216 ts_tablet_manager.cc:1428] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260925 22:52:10.003861 19063 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
I20260925 22:52:10.004026 19197 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
I20260925 22:52:10.030956 19221 raft_consensus.cc:482] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:10.031174 19221 raft_consensus.cc:504] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.032410 19221 leader_election.cc:288] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925), 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309)
I20260925 22:52:10.036124 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.033098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:33163 (local address 127.4.219.131:45309)
0925 22:52:10.033219 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.033223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.033240 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:52:10.033285 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.033288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.033325 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.033385 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:10.033651 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.034159 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.035175 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.035575 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.035704 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.035759 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.035945 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.036007 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:52:10.036144 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.032633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:45309 (local address 127.4.219.129:33163)
0925 22:52:10.033027 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.033043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.033156 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.033409 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.033412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.033426 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:10.033532 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.033536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.034271 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.034277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.035045 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.035054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.035704 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.035708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.035808 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.035832 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.035916 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.035987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":1,"thread_start_us":133,"threads_started":1}
I20260925 22:52:10.036415 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.032714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40925 (local address 127.4.219.129:36685)
0925 22:52:10.033036 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.033048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.033484 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.033750 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.033752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.033767 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:10.033887 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.033892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.034600 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.034605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.035362 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.035370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.035997 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.036001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.036106 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.036123 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.036209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.036282 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":82,"threads_started":1}
I20260925 22:52:10.036401 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.032755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:36685 (local address 127.4.219.130:40925)
0925 22:52:10.032901 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.032905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.033136 (+ 231us) server_negotiation.cc:407] Connection header received
0925 22:52:10.033606 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.033614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.033656 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.033755 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:10.033994 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.034488 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.035484 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.035880 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.036002 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.036046 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.036225 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.036289 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:52:10.037017 19151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0342786d717c4b98a057af9d7aac2b23" candidate_uuid: "f03679e2b39041099ebcf6bf8b05d180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803bce081f4946018d306cfa88c2751e" is_pre_election: true
I20260925 22:52:10.037199 19017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0342786d717c4b98a057af9d7aac2b23" candidate_uuid: "f03679e2b39041099ebcf6bf8b05d180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b35fc288894c20ab4f2793a8deb728" is_pre_election: true
I20260925 22:52:10.037354 19151 raft_consensus.cc:2454] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03679e2b39041099ebcf6bf8b05d180 in term 0.
I20260925 22:52:10.037487 19017 raft_consensus.cc:2454] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03679e2b39041099ebcf6bf8b05d180 in term 0.
I20260925 22:52:10.037885 18817 leader_election.cc:302] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728, f03679e2b39041099ebcf6bf8b05d180; no voters:
I20260925 22:52:10.038134 19221 raft_consensus.cc:2781] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:10.038214 19221 raft_consensus.cc:482] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:10.038257 19221 raft_consensus.cc:3037] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:10.039269 19221 raft_consensus.cc:504] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.039592 19221 leader_election.cc:288] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 1 election: Requested vote from peers d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925), 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309)
I20260925 22:52:10.039943 19017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0342786d717c4b98a057af9d7aac2b23" candidate_uuid: "f03679e2b39041099ebcf6bf8b05d180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b35fc288894c20ab4f2793a8deb728"
I20260925 22:52:10.039956 19151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0342786d717c4b98a057af9d7aac2b23" candidate_uuid: "f03679e2b39041099ebcf6bf8b05d180" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803bce081f4946018d306cfa88c2751e"
I20260925 22:52:10.040066 19151 raft_consensus.cc:3037] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:10.040066 19017 raft_consensus.cc:3037] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:10.041364 19151 raft_consensus.cc:2454] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03679e2b39041099ebcf6bf8b05d180 in term 1.
I20260925 22:52:10.041365 19017 raft_consensus.cc:2454] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03679e2b39041099ebcf6bf8b05d180 in term 1.
I20260925 22:52:10.041644 18819 leader_election.cc:302] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7b35fc288894c20ab4f2793a8deb728, f03679e2b39041099ebcf6bf8b05d180; no voters:
I20260925 22:52:10.041822 19221 raft_consensus.cc:2781] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:10.042058 19221 raft_consensus.cc:686] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 LEADER]: Becoming Leader. State: Replica: f03679e2b39041099ebcf6bf8b05d180, State: Running, Role: LEADER
I20260925 22:52:10.042387 19221 consensus_queue.cc:237] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.045603 18736 catalog_manager.cc:5495] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "f03679e2b39041099ebcf6bf8b05d180" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } health_report { overall_health: HEALTHY } } }
I20260925 22:52:10.061728 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.058450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.4.219.129:33529)
0925 22:52:10.058564 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.058567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.058582 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:52:10.058693 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.058695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.058749 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.058839 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:10.059019 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.059500 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.060332 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.060572 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.060649 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.060923 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:10.061347 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:10.061481 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.061578 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.061624 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:52:10.067688 19151 raft_consensus.cc:1261] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Refusing update from remote peer f03679e2b39041099ebcf6bf8b05d180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:10.067688 19017 raft_consensus.cc:1261] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Refusing update from remote peer f03679e2b39041099ebcf6bf8b05d180: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:10.068420 19221 consensus_queue.cc:1035] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:10.068643 19225 consensus_queue.cc:1035] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:10.077620 19228 mvcc.cc:205] Tried to move back new op lower bound from 7333383086350450688 to 7333383086256291840. Current Snapshot: MvccSnapshot[applied={T|T < 7333383086350450688 or (T in {7333383086350450688})}]
I20260925 22:52:10.078370 19230 mvcc.cc:205] Tried to move back new op lower bound from 7333383086350450688 to 7333383086256291840. Current Snapshot: MvccSnapshot[applied={T|T < 7333383086350450688 or (T in {7333383086350450688})}]
I20260925 22:52:10.083791 18738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:10.083976 18738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:10.086429 18738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Sending DeleteTablet for 3 replicas of tablet 0342786d717c4b98a057af9d7aac2b23
I20260925 22:52:10.087245 19131 tablet_service.cc:1514] Processing DeleteTablet for tablet 0342786d717c4b98a057af9d7aac2b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:10 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40712
I20260925 22:52:10.087251 18997 tablet_service.cc:1514] Processing DeleteTablet for tablet 0342786d717c4b98a057af9d7aac2b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:10 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:50848
I20260925 22:52:10.087384 18863 tablet_service.cc:1514] Processing DeleteTablet for tablet 0342786d717c4b98a057af9d7aac2b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:10 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:47502
I20260925 22:52:10.087852 19239 tablet_replica.cc:330] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: stopping tablet replica
I20260925 22:52:10.088029 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18800
I20260925 22:52:10.088047 19238 tablet_replica.cc:330] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: stopping tablet replica
I20260925 22:52:10.088388 19238 raft_consensus.cc:2229] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:10.088912 19238 raft_consensus.cc:2258] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:10.089324 19237 tablet_replica.cc:330] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: stopping tablet replica
I20260925 22:52:10.089605 19237 raft_consensus.cc:2229] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:10.089887 19237 raft_consensus.cc:2258] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:10.090654 19238 ts_tablet_manager.cc:1905] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:10.090999 19237 ts_tablet_manager.cc:1905] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:10.093088 19237 ts_tablet_manager.cc:1918] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:10.093197 19237 log.cc:1196] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/0342786d717c4b98a057af9d7aac2b23
I20260925 22:52:10.093513 19237 ts_tablet_manager.cc:1939] T 0342786d717c4b98a057af9d7aac2b23 P d7b35fc288894c20ab4f2793a8deb728: Deleting consensus metadata
I20260925 22:52:10.094097 19238 ts_tablet_manager.cc:1918] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:10.094199 19238 log.cc:1196] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/0342786d717c4b98a057af9d7aac2b23
I20260925 22:52:10.094605 19238 ts_tablet_manager.cc:1939] T 0342786d717c4b98a057af9d7aac2b23 P 803bce081f4946018d306cfa88c2751e: Deleting consensus metadata
I20260925 22:52:10.094667 18725 catalog_manager.cc:4841] TS d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): tablet 0342786d717c4b98a057af9d7aac2b23 (table test-table [id=2cf900cc5c08423db7c73c9b62b83146]) successfully deleted
I20260925 22:52:10.095783 18723 catalog_manager.cc:4841] TS 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): tablet 0342786d717c4b98a057af9d7aac2b23 (table test-table [id=2cf900cc5c08423db7c73c9b62b83146]) successfully deleted
W20260925 22:52:10.096369 18722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:33529 (error 108)
I20260925 22:52:10.096616 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18934
I20260925 22:52:10.098518 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.098136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33529 (local address 127.0.0.1:47522)
0925 22:52:10.098324 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.098406 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 22:52:10.098681 18722 catalog_manager.cc:4584] TS f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0342786d717c4b98a057af9d7aac2b23: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33529: connect: Connection refused (error 111)
I20260925 22:52:10.102505 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19068
I20260925 22:52:10.107856 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 18706
I20260925 22:52:10.114096 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--webserver_interface=127.4.219.190
--webserver_port=38627
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:10.222848 19241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:10.223165 19241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:10.223248 19241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:10.227311 19241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:10.227404 19241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:10.227443 19241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:10.227501 19241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:10.227540 19241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:10.232833 19241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:10.233911 19241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:10.234933 19241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:10.247865 19248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.247844 19247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.247870 19250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:10.248152 19241 server_base.cc:1034] running on GCE node
I20260925 22:52:10.248948 19241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:10.250168 19241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:10.251361 19241 hybrid_clock.cc:648] HybridClock initialized: now 1790376730251348 us; error 27 us; skew 500 ppm
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:10.254115 19241 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:10.254963 19241 webserver.cc:460] Webserver started at http://127.4.219.190:38627/ using document root <none> and password file <none>
I20260925 22:52:10.255523 19241 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:10.255595 19241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:10.259428 19241 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:10.261624 19257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.262840 19241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:10.262976 19241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "7eeae6a1d8fb468a997527d46ba4494e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "3204ee93757ad753dffec37122b49ebb"
server_key_iv: "4672436ddccd0f2583bcfb34797af9c1"
server_key_version: "encryptionkey@0"
I20260925 22:52:10.263561 19241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:10.276933 19241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:10.280310 19241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:10.280519 19241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:10.287618 19241 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39645
I20260925 22:52:10.287904 19309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39645 every 8 connection(s)
I20260925 22:52:10.288792 19241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:10.290153 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19241
I20260925 22:52:10.290611 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:33529
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=37627
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:10.293877 19310 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:52:10.295615 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap starting.
I20260925 22:52:10.312672 19310 log.cc:824] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:10.321087 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:10.321465 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap complete.
I20260925 22:52:10.325850 19310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:10.326287 19310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.326820 19310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:10.326968 19310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:10.327064 19310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:10.327179 19310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:52:10.329195 19310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:10.329545 19310 leader_election.cc:302] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeae6a1d8fb468a997527d46ba4494e; no voters:
I20260925 22:52:10.329887 19310 leader_election.cc:288] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:52:10.329977 19314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:52:10.330240 19314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 LEADER]: Becoming Leader. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Running, Role: LEADER
I20260925 22:52:10.330577 19314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:10.331156 19310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:10.332016 19315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:10.332132 19315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:10.332057 19316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeae6a1d8fb468a997527d46ba4494e. Latest consensus state: current_term: 2 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:10.332412 19316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:10.336206 19329 catalog_manager.cc:1259] Loaded cluster ID: 1044e499d6324c9db907a65d2c9658ce
I20260925 22:52:10.336293 19329 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: loading cluster ID for follower catalog manager: success
W20260925 22:52:10.337486 19329 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260925 22:52:10.337574 19329 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260925 22:52:10.337648 19323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:10.339499 19323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:10.339947 19323 catalog_manager.cc:1259] Loaded cluster ID: 1044e499d6324c9db907a65d2c9658ce
I20260925 22:52:10.340021 19323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:10.340948 19323 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:52:10.343657 19323 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:52:10.347990 19323 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:10.348830 19323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Loaded TSK: 0
I20260925 22:52:10.349939 19323 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:52:10.412199 19312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:10.412492 19312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:10.412595 19312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:10.417028 19312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:10.417091 19312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:10.417191 19312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:10.424593 19312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=37627
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:10.425765 19312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:10.426920 19312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:10.436493 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.436507 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.436625 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:10.436776 19312 server_base.cc:1034] running on GCE node
I20260925 22:52:10.437227 19312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:10.437896 19312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:10.439074 19312 hybrid_clock.cc:648] HybridClock initialized: now 1790376730439043 us; error 40 us; skew 500 ppm
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:10.441972 19312 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:10.442850 19312 webserver.cc:460] Webserver started at http://127.4.219.129:37627/ using document root <none> and password file <none>
I20260925 22:52:10.443418 19312 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:10.443521 19312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:10.447428 19312 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 22:52:10.449666 19347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.451017 19312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:52:10.451141 19312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "f03679e2b39041099ebcf6bf8b05d180"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9f30fc770c3a439f6591b83800d41ce2"
server_key_iv: "c3372c2a9532e1de466ed1b3a06d0afd"
server_key_version: "encryptionkey@0"
I20260925 22:52:10.451690 19312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:10.483031 19312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:10.486332 19312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:10.486557 19312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:10.487219 19312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:10.488368 19354 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:52:10.492403 19312 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:52:10.492491 19312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 22:52:10.492611 19312 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:52:10.494261 19312 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:52:10.494341 19312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:52:10.494737 19354 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap starting.
I20260925 22:52:10.504782 19312 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:33529
I20260925 22:52:10.504822 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:33529 every 8 connection(s)
I20260925 22:52:10.505735 19312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:10.508170 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19312
I20260925 22:52:10.508664 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:40925
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=37133
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:10.513681 19354 log.cc:824] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:10.521700 19354 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:10.522151 19354 tablet_bootstrap.cc:492] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap complete.
I20260925 22:52:10.523489 19354 ts_tablet_manager.cc:1397] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260925 22:52:10.526521 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.507823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:54655 (local address 127.4.219.190:39645)
0925 22:52:10.508487 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.508513 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.509457 (+ 944us) server_negotiation.cc:407] Connection header received
0925 22:52:10.511078 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.511089 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.511389 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.511647 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:10.513124 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.514843 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.516503 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.517348 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.517559 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.520693 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:10.520699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:10.520715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:10.520749 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:10.522960 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.523585 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.523591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.523596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.523752 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.524152 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.524155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.524157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.524644 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:10.524771 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.525421 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.525715 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":108,"threads_started":1}
I20260925 22:52:10.526521 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.507796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:54655)
0925 22:52:10.509254 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.509313 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.510565 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.511805 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.511814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.512300 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:10.512938 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.512957 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.515024 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.515028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.516365 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.516376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.517508 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.517513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.517690 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.518408 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:10.518426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:10.520462 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:10.523098 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.523104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.523117 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.523448 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.523853 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.523856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.523858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.524030 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.524771 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:10.524778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:10.525067 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.525255 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.525578 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":148,"threads_started":1}
I20260925 22:52:10.526427 19354 raft_consensus.cc:348] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.527235 19354 raft_consensus.cc:729] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03679e2b39041099ebcf6bf8b05d180, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.527786 19354 consensus_queue.cc:260] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.528847 19354 ts_tablet_manager.cc:1428] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260925 22:52:10.529546 19462 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:10.529774 19462 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:10.530308 19462 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
I20260925 22:52:10.532985 19273 ts_manager.cc:194] Registered new tserver with Master: f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529)
I20260925 22:52:10.536288 19273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:54655
I20260925 22:52:10.538419 19462 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:52:10.546552 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.535697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33529 (local address 127.0.0.1:47536)
0925 22:52:10.536454 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.536494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.536676 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.537311 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.537317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.537712 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:10.537832 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.537840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.539196 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.539199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.540114 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.540125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.540712 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.540716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.540822 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.541419 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:10.541433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:10.542927 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:10.544853 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.544863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.544868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.545090 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.545403 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.545408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.545410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.545494 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.545931 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:10.545937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:10.546058 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.546191 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.546376 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":102,"threads_started":1}
I20260925 22:52:10.546587 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.535838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.4.219.129:33529)
0925 22:52:10.536193 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.536204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.536612 (+ 408us) server_negotiation.cc:407] Connection header received
0925 22:52:10.536824 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.536829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.537031 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.537159 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:10.537977 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.539055 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.540237 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.540606 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.540690 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.543059 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:10.543070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:10.543086 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:10.543107 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:10.544730 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.545219 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.545224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.545225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.545306 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.545582 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.545587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.545589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.545817 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:10.545957 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.546290 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.546443 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":106,"threads_started":1}
I20260925 22:52:10.547501 19396 tablet_service.cc:1514] Processing DeleteTablet for tablet 0342786d717c4b98a057af9d7aac2b23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:10 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:47536
I20260925 22:52:10.548471 19471 tablet_replica.cc:330] stopping tablet replica
I20260925 22:52:10.548774 19471 raft_consensus.cc:2229] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:10.549060 19471 raft_consensus.cc:2258] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:10.550078 19471 ts_tablet_manager.cc:1905] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:10.553594 19471 ts_tablet_manager.cc:1918] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:10.553712 19471 log.cc:1196] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/0342786d717c4b98a057af9d7aac2b23
I20260925 22:52:10.553980 19471 ts_tablet_manager.cc:1939] T 0342786d717c4b98a057af9d7aac2b23 P f03679e2b39041099ebcf6bf8b05d180: Deleting consensus metadata
I20260925 22:52:10.555675 19258 catalog_manager.cc:4841] TS f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529): tablet 0342786d717c4b98a057af9d7aac2b23 (table test-table [id=2cf900cc5c08423db7c73c9b62b83146]) successfully deleted
W20260925 22:52:10.629812 19466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:10.630054 19466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:10.630095 19466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:10.634361 19466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:10.634416 19466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:10.634500 19466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:10.639776 19466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=37133
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:10.640898 19466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:10.641906 19466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:10.651621 19477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.651647 19479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:10.651949 19466 server_base.cc:1034] running on GCE node
W20260925 22:52:10.651618 19476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:10.652503 19466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:10.653185 19466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:10.654347 19466 hybrid_clock.cc:648] HybridClock initialized: now 1790376730654315 us; error 43 us; skew 500 ppm
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:10.657178 19466 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:10.658032 19466 webserver.cc:460] Webserver started at http://127.4.219.130:37133/ using document root <none> and password file <none>
I20260925 22:52:10.658600 19466 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:10.658704 19466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:10.662559 19466 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:52:10.664611 19486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.665773 19466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:10.665920 19466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "d7b35fc288894c20ab4f2793a8deb728"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9e0b1615c87602ec52b6f31266b29018"
server_key_iv: "1aab064b24ebfbe760d740448887b3b3"
server_key_version: "encryptionkey@0"
I20260925 22:52:10.666512 19466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:10.683030 19466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:10.685806 19466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:10.686002 19466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:10.686565 19466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:10.687403 19466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:10.687472 19466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.687543 19466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:10.687587 19466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.695783 19466 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40925
I20260925 22:52:10.695863 19599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40925 every 8 connection(s)
I20260925 22:52:10.696791 19466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:10.697105 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19466
I20260925 22:52:10.697561 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:45309
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=36927
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:10.711854 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.698766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:38243 (local address 127.4.219.190:39645)
0925 22:52:10.698898 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.698903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.699968 (+ 1065us) server_negotiation.cc:407] Connection header received
0925 22:52:10.701295 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.701299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.701370 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.701466 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:10.702598 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.703582 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.704778 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.705216 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.705351 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.708148 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:10.708154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:10.708159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:10.708195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:10.709852 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.710345 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.710348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.710350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.710433 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.710731 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.710734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.710736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.710948 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:10.711041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.711560 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.711684 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:52:10.712720 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.699182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:38243)
0925 22:52:10.699702 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.699758 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.701066 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.701635 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.701641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.701981 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:10.702441 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.702454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.703704 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.703707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.704651 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.704660 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.705394 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.705397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.705544 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.706178 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:10.706193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:10.707971 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:10.709968 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.709972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.709985 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.710253 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.710516 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.710518 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.710520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.710649 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.711049 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:10.711054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:10.711315 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.711517 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.711793 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260925 22:52:10.713914 19600 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:10.714156 19600 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:10.714685 19600 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
I20260925 22:52:10.715785 19273 ts_manager.cc:194] Registered new tserver with Master: d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
I20260925 22:52:10.716349 19273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:38243
W20260925 22:52:10.813377 19602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:10.813632 19602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:10.813673 19602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:10.817778 19602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:10.817835 19602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:10.817935 19602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:10.823074 19602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:45309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=36927
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:10.824115 19602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:10.825177 19602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:52:10.835247 19602 server_base.cc:1034] running on GCE node
W20260925 22:52:10.835196 19609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.835207 19608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:10.835196 19611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:10.835877 19602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:10.836524 19602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:10.837749 19602 hybrid_clock.cc:648] HybridClock initialized: now 1790376730837729 us; error 30 us; skew 500 ppm
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:10.840420 19602 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:10.841271 19602 webserver.cc:460] Webserver started at http://127.4.219.131:36927/ using document root <none> and password file <none>
I20260925 22:52:10.841833 19602 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:10.841938 19602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:10.845772 19602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:52:10.847800 19618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.849210 19602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:10.849354 19602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "803bce081f4946018d306cfa88c2751e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "50c82170319ee01c97e068774e71ba77"
server_key_iv: "56af342dfc7503d1173691f60bc0066c"
server_key_version: "encryptionkey@0"
I20260925 22:52:10.849972 19602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:10.875885 19602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:10.878772 19602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:10.878976 19602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:10.879595 19602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:10.880460 19602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:10.880534 19602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.880658 19602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:10.880712 19602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:10.889653 19602 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:45309
I20260925 22:52:10.889675 19731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:45309 every 8 connection(s)
I20260925 22:52:10.890671 19602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:10.894515 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19602
Sep 25 22:52:10 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376730, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:10.903538 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.895153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.4.219.190:39645)
0925 22:52:10.895413 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.895418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.895636 (+ 218us) server_negotiation.cc:407] Connection header received
0925 22:52:10.895807 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.895815 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.895866 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.895954 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:10.896689 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.897197 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.898160 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.898386 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.898492 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.899880 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:10.899884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:10.899885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:10.899906 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:10.901894 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.902304 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.902309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.902311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.902399 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.902683 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.902687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.902689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.902875 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:10.903086 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.903225 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.903340 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":96,"threads_started":1}
I20260925 22:52:10.904829 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.892428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45635 (local address 127.4.219.190:39645)
0925 22:52:10.892619 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.892622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.893477 (+ 855us) server_negotiation.cc:407] Connection header received
0925 22:52:10.894812 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.894816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.894873 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.894953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:10.896172 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.896768 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.898014 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.898226 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.898342 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.901066 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:10.901073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:10.901076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:10.901118 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:10.902564 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.903068 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.903078 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.903080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.903148 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:10.903484 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:10.903489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:10.903491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:10.903688 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:10.903781 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.904524 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.904677 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:52:10.905647 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.892887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:45635)
0925 22:52:10.893325 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.893372 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.894571 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.895125 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.895135 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.895552 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:10.896035 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.896047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.896890 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.896893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.897887 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.897897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.898370 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.898374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.898541 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.899245 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:10.899260 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:10.900877 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:10.902694 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.902700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.902720 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.902958 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.903248 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:10.903256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:10.903257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:10.903388 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:10.903806 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:10.903812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:10.904078 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.904287 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.904630 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260925 22:52:10.906744 19732 heartbeater.cc:347] Connected to a master server at 127.4.219.190:39645
I20260925 22:52:10.906946 19732 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:10.907395 19732 heartbeater.cc:511] Master 127.4.219.190:39645 requested a full tablet report, sending...
I20260925 22:52:10.908306 19273 ts_manager.cc:194] Registered new tserver with Master: 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309)
I20260925 22:52:10.908895 19273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:45635
I20260925 22:52:10.917476 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:52:10.921998 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.918137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.4.219.190:39645)
0925 22:52:10.918343 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.918347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.918474 (+ 127us) server_negotiation.cc:407] Connection header received
0925 22:52:10.918687 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.918689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.918794 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.918910 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:10.919157 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.919663 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.920573 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.920800 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.920912 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.921224 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:10.921614 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:10.921745 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.921806 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.921856 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:52:10.922616 19273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:52:10.925853 19273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:52:10.933750 19396 tablet_service.cc:1467] Processing CreateTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 (DEFAULT_TABLE table=test-table [id=1dc42fdd1a1d42c7b55a0151fd676243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:10.934466 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cb14d326cb46fa88cc8db2724a8563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:10.937415 19354 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap starting.
I20260925 22:52:10.937521 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.932284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:45309 (local address 127.0.0.1:40728)
0925 22:52:10.932384 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.932400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.932477 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.933508 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.933510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.933529 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:10.933635 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.933641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.934850 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.934856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.935920 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.935929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.937083 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.937087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.937177 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.937219 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.937303 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.937390 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 22:52:10.938022 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.932591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.4.219.131:45309)
0925 22:52:10.932963 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.932977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.933004 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:52:10.933089 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.933095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.933338 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.933538 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:10.933736 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.934711 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.936053 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.936963 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.937090 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.937239 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.937344 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.937436 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":172,"threads_started":1}
I20260925 22:52:10.938330 19354 tablet_bootstrap.cc:654] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:10.938361 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.932788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40925 (local address 127.0.0.1:50856)
0925 22:52:10.933001 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:10.933015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:10.933106 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:10.933988 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:10.933992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:10.934011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:10.934099 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.934105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.935344 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.935348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.936660 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:10.936671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.937901 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.937905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:10.937995 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.938017 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:52:10.938117 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:52:10.938183 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260925 22:52:10.938547 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:10.933107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.4.219.130:40925)
0925 22:52:10.933517 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:10.933529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:10.933552 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:52:10.933625 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:10.933630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:10.933838 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:10.934043 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:10.934208 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.935200 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.936792 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:10.937593 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:10.937692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:10.937838 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:10.938223 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 22:52:10.938326 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":108,"threads_started":1}
I20260925 22:52:10.940210 19354 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: No bootstrap required, opened a new log
I20260925 22:52:10.940295 19354 ts_tablet_manager.cc:1397] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:52:10.940752 19354 raft_consensus.cc:348] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.940871 19354 raft_consensus.cc:374] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.940907 19354 raft_consensus.cc:729] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03679e2b39041099ebcf6bf8b05d180, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.941058 19354 consensus_queue.cc:260] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.941313 19354 ts_tablet_manager.cc:1428] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:52:10.941452 19666 tablet_service.cc:1467] Processing CreateTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 (DEFAULT_TABLE table=test-table [id=1dc42fdd1a1d42c7b55a0151fd676243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:10.941519 19534 tablet_service.cc:1467] Processing CreateTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 (DEFAULT_TABLE table=test-table [id=1dc42fdd1a1d42c7b55a0151fd676243]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:10.942483 19666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cb14d326cb46fa88cc8db2724a8563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:10.942534 19534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cb14d326cb46fa88cc8db2724a8563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:10.949390 19745 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Bootstrap starting.
I20260925 22:52:10.949899 19746 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Bootstrap starting.
I20260925 22:52:10.952168 19745 tablet_bootstrap.cc:654] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:10.952373 19746 tablet_bootstrap.cc:654] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:10.953141 19746 log.cc:824] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:10.953141 19745 log.cc:824] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:10.955380 19746 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: No bootstrap required, opened a new log
I20260925 22:52:10.955579 19746 ts_tablet_manager.cc:1397] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:52:10.955637 19745 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: No bootstrap required, opened a new log
I20260925 22:52:10.955799 19745 ts_tablet_manager.cc:1397] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:52:10.959672 19746 raft_consensus.cc:348] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.959672 19745 raft_consensus.cc:348] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.959985 19745 raft_consensus.cc:374] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.959985 19746 raft_consensus.cc:374] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:10.960040 19745 raft_consensus.cc:729] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b35fc288894c20ab4f2793a8deb728, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.960040 19746 raft_consensus.cc:729] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803bce081f4946018d306cfa88c2751e, State: Initialized, Role: FOLLOWER
I20260925 22:52:10.960577 19746 consensus_queue.cc:260] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.960577 19745 consensus_queue.cc:260] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:10.961287 19600 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
I20260925 22:52:10.961495 19732 heartbeater.cc:503] Master 127.4.219.190:39645 was elected leader, sending a full tablet report...
I20260925 22:52:10.961660 19746 ts_tablet_manager.cc:1428] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:52:10.961660 19745 ts_tablet_manager.cc:1428] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 22:52:11.006982 19463 tablet.cc:2369] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:11.106350 19749 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:11.106606 19749 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:11.107895 19749 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529)
I20260925 22:52:11.112123 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.108789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:47961 (local address 127.4.219.131:45309)
0925 22:52:11.108953 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:11.108957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:11.108970 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:52:11.109020 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:11.109022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:11.109061 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:11.109161 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:11.109502 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.110009 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111099 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.111508 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111640 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:11.111703 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:11.111931 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 22:52:11.112000 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:52:11.112133 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.108486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:37159 (local address 127.4.219.129:33529)
0925 22:52:11.108911 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:11.108916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:11.108933 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:52:11.108993 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:11.108996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:11.109063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:11.109174 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:11.109494 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.110012 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111102 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.111542 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111678 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:11.111854 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:11.111923 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 22:52:11.111996 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":101,"threads_started":1}
I20260925 22:52:11.112126 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.108148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:45309 (local address 127.4.219.130:47961)
0925 22:52:11.108293 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.108309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:11.108414 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:11.109185 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:11.109188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:11.109225 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:11.109380 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:11.109386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.110124 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.110127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:11.110977 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:11.110987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.111636 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:11.111774 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:11.111827 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 22:52:11.111917 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:52:11.111988 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":129,"spinlock_wait_cycles":3456}
I20260925 22:52:11.112142 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.108209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33529 (local address 127.4.219.130:37159)
0925 22:52:11.108804 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.108818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:11.108909 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:11.109182 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:11.109185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:11.109214 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:11.109372 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:11.109378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.110123 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.110129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:11.110969 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:11.110977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.111665 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.111668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:11.111811 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:11.111833 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:52:11.111909 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:52:11.111985 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":65,"spinlock_wait_cycles":15488,"thread_start_us":161,"threads_started":1}
I20260925 22:52:11.112870 19686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cb14d326cb46fa88cc8db2724a8563" candidate_uuid: "d7b35fc288894c20ab4f2793a8deb728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803bce081f4946018d306cfa88c2751e" is_pre_election: true
I20260925 22:52:11.113145 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cb14d326cb46fa88cc8db2724a8563" candidate_uuid: "d7b35fc288894c20ab4f2793a8deb728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03679e2b39041099ebcf6bf8b05d180" is_pre_election: true
I20260925 22:52:11.113222 19686 raft_consensus.cc:2454] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b35fc288894c20ab4f2793a8deb728 in term 0.
I20260925 22:52:11.113415 19415 raft_consensus.cc:2454] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b35fc288894c20ab4f2793a8deb728 in term 0.
I20260925 22:52:11.113736 19488 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728; no voters:
I20260925 22:52:11.114049 19749 raft_consensus.cc:2781] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:11.114133 19749 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:11.114184 19749 raft_consensus.cc:3037] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:11.115231 19749 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:11.115615 19749 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [CANDIDATE]: Term 1 election: Requested vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529)
I20260925 22:52:11.116001 19686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cb14d326cb46fa88cc8db2724a8563" candidate_uuid: "d7b35fc288894c20ab4f2793a8deb728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803bce081f4946018d306cfa88c2751e"
I20260925 22:52:11.116001 19415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cb14d326cb46fa88cc8db2724a8563" candidate_uuid: "d7b35fc288894c20ab4f2793a8deb728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03679e2b39041099ebcf6bf8b05d180"
I20260925 22:52:11.116135 19415 raft_consensus.cc:3037] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:11.116137 19686 raft_consensus.cc:3037] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:11.117499 19686 raft_consensus.cc:2454] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b35fc288894c20ab4f2793a8deb728 in term 1.
I20260925 22:52:11.117499 19415 raft_consensus.cc:2454] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b35fc288894c20ab4f2793a8deb728 in term 1.
I20260925 22:52:11.117769 19488 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728; no voters:
I20260925 22:52:11.117933 19749 raft_consensus.cc:2781] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:11.118137 19749 raft_consensus.cc:686] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 1 LEADER]: Becoming Leader. State: Replica: d7b35fc288894c20ab4f2793a8deb728, State: Running, Role: LEADER
I20260925 22:52:11.118486 19749 consensus_queue.cc:237] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:11.121680 19273 catalog_manager.cc:5495] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130). New cstate: current_term: 1 leader_uuid: "d7b35fc288894c20ab4f2793a8deb728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } health_report { overall_health: UNKNOWN } } }
W20260925 22:52:11.141629 19733 tablet.cc:2369] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:11.177093 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.173711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.4.219.130:40925)
0925 22:52:11.173859 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:11.173863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:11.173896 (+ 33us) server_negotiation.cc:407] Connection header received
0925 22:52:11.173976 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:11.173978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:11.174034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:11.174117 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:11.174323 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.174824 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.175657 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:11.175865 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:11.175941 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:11.176193 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:11.176628 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:11.176717 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:11.176850 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 22:52:11.176921 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:52:11.182967 19415 raft_consensus.cc:1261] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Refusing update from remote peer d7b35fc288894c20ab4f2793a8deb728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:11.182971 19686 raft_consensus.cc:1261] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Refusing update from remote peer d7b35fc288894c20ab4f2793a8deb728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:11.183615 19749 consensus_queue.cc:1035] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:11.183851 19753 consensus_queue.cc:1035] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:11.190119 19757 mvcc.cc:205] Tried to move back new op lower bound from 7333383090918330368 to 7333383090663927808. Current Snapshot: MvccSnapshot[applied={T|T < 7333383090918330368}]
I20260925 22:52:11.192324 19756 mvcc.cc:205] Tried to move back new op lower bound from 7333383090918330368 to 7333383090663927808. Current Snapshot: MvccSnapshot[applied={T|T < 7333383090918330368 or (T in {7333383090918330368})}]
I20260925 22:52:11.197671 19274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:11.197887 19274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:11.200309 19274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Sending DeleteTablet for 3 replicas of tablet 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:11.201092 19534 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:50856
I20260925 22:52:11.201162 19396 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:47536
I20260925 22:52:11.201170 19666 tablet_service.cc:1514] Processing DeleteTablet for tablet 70cb14d326cb46fa88cc8db2724a8563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:11 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40728
I20260925 22:52:11.201546 19766 tablet_replica.cc:330] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: stopping tablet replica
I20260925 22:52:11.201582 19767 tablet_replica.cc:330] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: stopping tablet replica
I20260925 22:52:11.201655 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19312
I20260925 22:52:11.202317 19767 raft_consensus.cc:2229] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:11.202648 19767 raft_consensus.cc:2258] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:11.204145 19767 ts_tablet_manager.cc:1905] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:11.205232 19768 tablet_replica.cc:330] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: stopping tablet replica
I20260925 22:52:11.205595 19768 raft_consensus.cc:2229] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:52:11.206035 19768 raft_consensus.cc:2258] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:11.207350 19767 ts_tablet_manager.cc:1918] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:11.207468 19767 log.cc:1196] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:11.207502 19768 ts_tablet_manager.cc:1905] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:11.207856 19767 ts_tablet_manager.cc:1939] T 70cb14d326cb46fa88cc8db2724a8563 P 803bce081f4946018d306cfa88c2751e: Deleting consensus metadata
I20260925 22:52:11.209139 19259 catalog_manager.cc:4841] TS 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): tablet 70cb14d326cb46fa88cc8db2724a8563 (table test-table [id=1dc42fdd1a1d42c7b55a0151fd676243]) successfully deleted
I20260925 22:52:11.210362 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19466
W20260925 22:52:11.210434 19258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:33529 (error 108)
I20260925 22:52:11.211781 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.211494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33529 (local address 127.0.0.1:47550)
0925 22:52:11.211623 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.211708 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:52:11.212008 19258 catalog_manager.cc:4584] TS f03679e2b39041099ebcf6bf8b05d180 (127.4.219.129:33529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70cb14d326cb46fa88cc8db2724a8563: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33529: connect: Connection refused (error 111)
I20260925 22:52:11.216796 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19602
I20260925 22:52:11.217015 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.216735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40925 (local address 127.0.0.1:50866)
0925 22:52:11.216863 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.216923 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.130:40925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 22:52:11.222966 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19241
I20260925 22:52:11.229744 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--webserver_interface=127.4.219.190
--webserver_port=38627
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:11.339334 19770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:11.339617 19770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:11.339658 19770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:11.343745 19770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:11.343822 19770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:11.343842 19770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:11.343858 19770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:11.343874 19770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:11.349431 19770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=38627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:11.350575 19770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:11.351631 19770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:11.360114 19776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:11.360116 19775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:11.360317 19770 server_base.cc:1034] running on GCE node
W20260925 22:52:11.360069 19778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:11.361025 19770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:11.362221 19770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:11.363405 19770 hybrid_clock.cc:648] HybridClock initialized: now 1790376731363371 us; error 48 us; skew 500 ppm
Sep 25 22:52:11 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376731, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:11.366220 19770 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:11.367085 19770 webserver.cc:460] Webserver started at http://127.4.219.190:38627/ using document root <none> and password file <none>
I20260925 22:52:11.367651 19770 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:11.367745 19770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:11.371580 19770 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:11.374106 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:11.375455 19770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:52:11.375560 19770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "7eeae6a1d8fb468a997527d46ba4494e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "3204ee93757ad753dffec37122b49ebb"
server_key_iv: "4672436ddccd0f2583bcfb34797af9c1"
server_key_version: "encryptionkey@0"
I20260925 22:52:11.376143 19770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:11.394420 19770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:11.397800 19770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:11.397961 19770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:11.404798 19770 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39645
I20260925 22:52:11.405048 19837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39645 every 8 connection(s)
I20260925 22:52:11.405956 19770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:11.410215 19838 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:52:11.411804 19838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap starting.
I20260925 22:52:11.415992 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19770
I20260925 22:52:11.416464 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:33529
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=37627
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:11.430173 19838 log.cc:824] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:11.441483 19838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:11.441874 19838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e: Bootstrap complete.
I20260925 22:52:11.445973 19838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:11.446365 19838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Initialized, Role: FOLLOWER
I20260925 22:52:11.446866 19838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:11.446981 19838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:11.447069 19838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:11.447187 19838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:52:11.449344 19838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:11.449688 19838 leader_election.cc:302] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eeae6a1d8fb468a997527d46ba4494e; no voters:
I20260925 22:52:11.450059 19838 leader_election.cc:288] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:52:11.450166 19842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:52:11.450434 19842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [term 3 LEADER]: Becoming Leader. State: Replica: 7eeae6a1d8fb468a997527d46ba4494e, State: Running, Role: LEADER
I20260925 22:52:11.450760 19842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } }
I20260925 22:52:11.451309 19838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:11.451761 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eeae6a1d8fb468a997527d46ba4494e. Latest consensus state: current_term: 3 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:11.451882 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:11.452188 19843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7eeae6a1d8fb468a997527d46ba4494e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eeae6a1d8fb468a997527d46ba4494e" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39645 } } }
I20260925 22:52:11.452302 19843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eeae6a1d8fb468a997527d46ba4494e [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:11.453742 19846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:11.458670 19846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:11.459369 19846 catalog_manager.cc:1259] Loaded cluster ID: 1044e499d6324c9db907a65d2c9658ce
I20260925 22:52:11.459431 19846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:52:11.460446 19846 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f58597189ad google::LogMessage::Fail() at ??:0
@ 0x7f585971b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f58597184de google::LogMessage::Flush() at ??:0
@ 0x7f5859719a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5864057fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5864058d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5864055579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5864099640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f585b96a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f585b96ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f585b96c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f585b95b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f585cf9f6db start_thread at ??:0
@ 0x7f585821661f clone at ??:0
W20260925 22:52:11.554090 19840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:11.554363 19840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:11.554409 19840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:11.558493 19840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:11.558544 19840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:11.558622 19840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:11.564031 19840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=37627
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:11.565142 19840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:11.566128 19840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:11.575649 19880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:11.575755 19882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:11.575649 19879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:11.576485 19840 server_base.cc:1034] running on GCE node
I20260925 22:52:11.576948 19840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:11.577703 19840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:11.578936 19840 hybrid_clock.cc:648] HybridClock initialized: now 1790376731578895 us; error 51 us; skew 500 ppm
Sep 25 22:52:11 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376731, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:11.582042 19840 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:11.583103 19840 webserver.cc:460] Webserver started at http://127.4.219.129:37627/ using document root <none> and password file <none>
I20260925 22:52:11.583824 19840 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:11.583899 19840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:11.589383 19840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:52:11.592381 19889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:11.593991 19840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:11.594115 19840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "f03679e2b39041099ebcf6bf8b05d180"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9f30fc770c3a439f6591b83800d41ce2"
server_key_iv: "c3372c2a9532e1de466ed1b3a06d0afd"
server_key_version: "encryptionkey@0"
I20260925 22:52:11.594884 19840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:11.636409 19840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:11.640208 19840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:11.640434 19840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:11.641134 19840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:11.642405 19896 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:52:11.648334 19840 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:52:11.648445 19840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 22:52:11.648552 19840 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:52:11.650516 19840 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:52:11.650593 19840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:52:11.650914 19896 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap starting.
I20260925 22:52:11.661417 19840 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:33529
I20260925 22:52:11.662411 19840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:11.662977 20003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:33529 every 8 connection(s)
I20260925 22:52:11.664106 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 19840
I20260925 22:52:11.664575 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:40925
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=37133
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:11.673193 19896 log.cc:824] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:11.682161 19896 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:11.682588 19896 tablet_bootstrap.cc:492] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Bootstrap complete.
I20260925 22:52:11.684020 19896 ts_tablet_manager.cc:1397] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260925 22:52:11.687377 19896 raft_consensus.cc:348] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:11.687968 19896 raft_consensus.cc:729] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03679e2b39041099ebcf6bf8b05d180, State: Initialized, Role: FOLLOWER
I20260925 22:52:11.688516 19896 consensus_queue.cc:260] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:11.689612 19896 ts_tablet_manager.cc:1428] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
W20260925 22:52:11.706091 20006 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:52:11.664811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:41307)
0925 22:52:11.665388 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.665461 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:11.666665 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:11.705325 (+ 38660us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":152,"threads_started":1}
W20260925 22:52:11.706763 20004 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:39645 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:39645: Client connection negotiation failed: client connection to 127.4.219.190:39645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:52:11.707545 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.707262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:44529)
0925 22:52:11.707418 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.707493 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 22:52:11.708217 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.708013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:40661)
0925 22:52:11.708119 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.708163 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:52:11.708428 20004 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:52:11.785605 20007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:11.785918 20007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:11.785995 20007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:11.790015 20007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:11.790105 20007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:11.790196 20007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:11.795562 20007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:40925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=37133
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:11.796974 20007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:11.798000 20007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:52:11.807873 20007 server_base.cc:1034] running on GCE node
W20260925 22:52:11.807837 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:11.807837 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:11.807847 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:11.808597 20007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:11.809247 20007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:11.810436 20007 hybrid_clock.cc:648] HybridClock initialized: now 1790376731810409 us; error 49 us; skew 500 ppm
Sep 25 22:52:11 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376731, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:11.813205 20007 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:11.814037 20007 webserver.cc:460] Webserver started at http://127.4.219.130:37133/ using document root <none> and password file <none>
I20260925 22:52:11.814587 20007 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:11.814677 20007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:11.818408 20007 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:11.820641 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:11.821906 20007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:11.822031 20007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "d7b35fc288894c20ab4f2793a8deb728"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "9e0b1615c87602ec52b6f31266b29018"
server_key_iv: "1aab064b24ebfbe760d740448887b3b3"
server_key_version: "encryptionkey@0"
I20260925 22:52:11.822556 20007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:11.845820 20007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:11.848896 20007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:11.849123 20007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:11.849802 20007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:11.850891 20031 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 22:52:11.854774 20031 ts_tablet_manager.cc:1852] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 22:52:11.855154 20031 ts_tablet_manager.cc:1905] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:11.858904 20031 ts_tablet_manager.cc:1918] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:11.859035 20031 log.cc:1196] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:11.859370 20031 ts_tablet_manager.cc:1939] T 70cb14d326cb46fa88cc8db2724a8563 P d7b35fc288894c20ab4f2793a8deb728: Deleting consensus metadata
I20260925 22:52:11.859676 20007 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 22:52:11.859752 20007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260925 22:52:11.859845 20007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:11.859895 20007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:11.868846 20007 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:40925
I20260925 22:52:11.868889 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:40925 every 8 connection(s)
I20260925 22:52:11.869724 20007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:11.872627 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20007
I20260925 22:52:11.873072 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:45309
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:39645
--webserver_port=36927
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:42825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:11.873377 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.871798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:48641)
0925 22:52:11.872233 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.872377 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":109,"threads_started":1}
W20260925 22:52:11.873787 20139 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:39645 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:39645: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
I20260925 22:52:11.874419 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.874129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:36855)
0925 22:52:11.874269 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.874337 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 22:52:11.875085 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:11.874901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:40789)
0925 22:52:11.874999 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:11.875039 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:52:11.875259 20139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:52:11.981369 20142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:11.981662 20142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:11.981738 20142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:11.985754 20142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:11.985852 20142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:11.985941 20142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:11.991314 20142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:42825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:45309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=36927
--tserver_master_addrs=127.4.219.190:39645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:11.992852 20142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:11.994179 20142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:12.003994 20148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:12.004115 20142 server_base.cc:1034] running on GCE node
W20260925 22:52:12.004006 20147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:12.004319 20150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:12.004737 20142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:12.005388 20142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:12.006572 20142 hybrid_clock.cc:648] HybridClock initialized: now 1790376732006540 us; error 43 us; skew 500 ppm
Sep 25 22:52:12 dist-test-slave-kb1g krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376732, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:12.009258 20142 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:12.010128 20142 webserver.cc:460] Webserver started at http://127.4.219.131:36927/ using document root <none> and password file <none>
I20260925 22:52:12.010639 20142 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:12.010697 20142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:12.014408 20142 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:52:12.016295 20157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:12.017521 20142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:12.017645 20142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "803bce081f4946018d306cfa88c2751e"
format_stamp: "Formatted at 2026-09-25 22:52:09 on dist-test-slave-kb1g"
server_key: "50c82170319ee01c97e068774e71ba77"
server_key_iv: "56af342dfc7503d1173691f60bc0066c"
server_key_version: "encryptionkey@0"
I20260925 22:52:12.018170 20142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:12.042272 20142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:12.045655 20142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:12.045871 20142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:12.046473 20142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:12.047300 20142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:12.047346 20142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:12.047415 20142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:12.047452 20142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:12.056389 20142 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:45309
I20260925 22:52:12.056412 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:45309 every 8 connection(s)
I20260925 22:52:12.057461 20142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:12.059828 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20142
I20260925 22:52:12.061769 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:12.059616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:60771)
0925 22:52:12.060245 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:12.060408 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":175,"threads_started":1}
W20260925 22:52:12.062222 20271 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:39645 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:39645: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
I20260925 22:52:12.062762 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:12.062579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:52303)
0925 22:52:12.062672 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:12.062715 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 22:52:12.063365 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:12.063194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:47185)
0925 22:52:12.063278 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:12.063315 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:52:12.063577 20271 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:52:12.709476 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:12.709011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:46467)
0925 22:52:12.709312 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:12.709369 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
I20260925 22:52:12.876217 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:12.875808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:35087)
0925 22:52:12.876069 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:12.876129 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260925 22:52:13.064515 20277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.064112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:57051)
0925 22:52:13.064360 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:13.064420 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260925 22:52:13.479238 20278 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:13.479473 20278 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:13.480762 20278 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
Sep 25 22:52:13 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376731, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.131@KRBTEST.COM
Sep 25 22:52:13 dist-test-slave-kb1g krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376731, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.130@KRBTEST.COM
I20260925 22:52:13.495765 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.481198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:48605 (local address 127.4.219.131:45309)
0925 22:52:13.481680 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:13.481708 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:13.481742 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:52:13.482419 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:13.482425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:13.482772 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:13.483085 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:13.484404 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.485548 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.486713 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.487601 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.487757 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:13.490918 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:13.490924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:13.490937 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:13.490968 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:13.493199 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:13.493759 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:13.493765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:13.493771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:13.493905 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:13.494253 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:13.494258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:13.494260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:13.494671 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:13.494809 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:13.495318 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 22:52:13.495564 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":106,"threads_started":1}
I20260925 22:52:13.495754 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.481001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:45309 (local address 127.4.219.129:48605)
0925 22:52:13.481353 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:13.481370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:13.481477 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:13.483149 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:13.483156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:13.483721 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:13.484198 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:13.484214 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.485687 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.485690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:13.486586 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:13.486596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.487714 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.487718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:13.487846 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:13.488649 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:13.488667 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:13.490662 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:13.493325 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:13.493330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:13.493342 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:13.493591 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:13.494006 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:13.494009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:13.494011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:13.494150 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:13.494797 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:13.494803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:13.495015 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 22:52:13.495204 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:52:13.495537 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":121,"spinlock_wait_cycles":24448,"thread_start_us":83,"threads_started":1}
I20260925 22:52:13.495868 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.481001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:40925 (local address 127.4.219.129:54107)
0925 22:52:13.481441 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:13.481456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:13.481576 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:13.483149 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:13.483156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:13.483701 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:13.484199 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:13.484214 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.485546 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.485550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:13.486626 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:13.486638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.487590 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.487593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:13.487842 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:13.488522 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:13.488575 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:13.491056 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:13.493629 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:13.493635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:13.493637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:13.493851 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:13.494225 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:13.494230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:13.494232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:13.494323 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:13.494924 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:13.494928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:13.495035 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 22:52:13.495204 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 22:52:13.495626 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":408,"spinlock_wait_cycles":11520,"thread_start_us":61,"threads_started":1}
I20260925 22:52:13.495834 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.481256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:54107 (local address 127.4.219.130:40925)
0925 22:52:13.481657 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:13.481669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:13.481703 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:52:13.482422 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:13.482427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:13.482778 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:13.483084 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:13.484372 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.485399 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.486749 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:13.487449 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:13.487610 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:13.491315 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:13.491326 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:13.491344 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:13.491389 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:13.493503 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:13.493979 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:13.493988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:13.493998 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:13.494110 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:13.494428 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:13.494434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:13.494440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:13.494809 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:13.494956 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:13.495332 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 22:52:13.495625 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":83,"threads_started":1}
W20260925 22:52:13.498988 19891 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
W20260925 22:52:13.499235 19893 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:13.499318 19893 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03679e2b39041099ebcf6bf8b05d180; no voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728
I20260925 22:52:13.499632 20278 raft_consensus.cc:2726] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:52:13.710578 20280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.710280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:41177)
0925 22:52:13.710428 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:13.710494 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:52:13.877394 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:13.876982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:49363)
0925 22:52:13.877222 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:13.877278 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260925 22:52:14.065625 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:14.065252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:52839)
0925 22:52:14.065474 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:14.065525 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260925 22:52:14.711789 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:14.711352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:40323)
0925 22:52:14.711615 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:14.711676 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":74,"threads_started":1}
I20260925 22:52:14.878563 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:14.878141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:37095)
0925 22:52:14.878406 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:14.878459 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":67,"threads_started":1}
I20260925 22:52:15.039180 20287 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:15.039327 20287 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:15.039870 20287 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
W20260925 22:52:15.040752 19893 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
W20260925 22:52:15.040877 19891 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:15.040975 19891 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03679e2b39041099ebcf6bf8b05d180; no voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728
I20260925 22:52:15.041159 20287 raft_consensus.cc:2726] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:52:15.066677 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:15.066302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:56771)
0925 22:52:15.066540 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:15.066594 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260925 22:52:15.713018 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:15.712626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:48589)
0925 22:52:15.712864 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:15.712919 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260925 22:52:15.879738 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:15.879311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:54483)
0925 22:52:15.879577 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:15.879632 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260925 22:52:16.067821 20291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:16.067443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:54845)
0925 22:52:16.067679 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:16.067735 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260925 22:52:16.714474 20292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:16.713970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:58373)
0925 22:52:16.714313 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:16.714372 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":77,"threads_started":1}
I20260925 22:52:16.881001 20293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:16.880518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:36781)
0925 22:52:16.880849 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:16.880905 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":95,"threads_started":1}
I20260925 22:52:17.030905 20294 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:17.031032 20294 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:17.031495 20294 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
W20260925 22:52:17.032389 19893 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
W20260925 22:52:17.032500 19891 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:17.032603 19891 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03679e2b39041099ebcf6bf8b05d180; no voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728
I20260925 22:52:17.032744 20294 raft_consensus.cc:2726] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:52:17.069056 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:17.068653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:43971)
0925 22:52:17.068906 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:17.068965 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260925 22:52:17.715667 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:17.715290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:38201)
0925 22:52:17.715522 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:17.715573 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260925 22:52:17.882246 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:17.881784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:49781)
0925 22:52:17.882055 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:17.882155 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260925 22:52:18.070127 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:18.069747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:47543)
0925 22:52:18.069981 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:18.070035 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260925 22:52:18.716872 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:18.716414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:57033)
0925 22:52:18.716701 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:18.716754 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
I20260925 22:52:18.883596 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:18.883093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:45715)
0925 22:52:18.883442 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:18.883499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260925 22:52:19.071316 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:19.070859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:42251)
0925 22:52:19.071152 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:19.071208 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":72,"threads_started":1}
I20260925 22:52:19.718132 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:19.717726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:53371)
0925 22:52:19.717979 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:19.718036 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260925 22:52:19.884918 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:19.884445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:51357)
0925 22:52:19.884759 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:19.884818 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
I20260925 22:52:20.072482 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:20.072079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:45247)
0925 22:52:20.072324 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:20.072384 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260925 22:52:20.719403 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:20.718976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:47359)
0925 22:52:20.719250 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:20.719307 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260925 22:52:20.886049 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:20.885644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:59959)
0925 22:52:20.885903 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:20.885957 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260925 22:52:21.073658 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:21.073258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:54861)
0925 22:52:21.073512 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:21.073566 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260925 22:52:21.188606 20308 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:21.188745 20308 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:21.189278 20308 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
W20260925 22:52:21.190231 19893 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
W20260925 22:52:21.190325 19891 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:21.190389 19891 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03679e2b39041099ebcf6bf8b05d180; no voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728
I20260925 22:52:21.190598 20308 raft_consensus.cc:2726] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:52:21.720925 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:21.720359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:60949)
0925 22:52:21.720699 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:21.720762 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":95,"threads_started":1}
I20260925 22:52:21.887305 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:21.886876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:38825)
0925 22:52:21.887134 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:21.887207 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260925 22:52:22.075019 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:22.074601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:34211)
0925 22:52:22.074863 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:22.074929 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260925 22:52:22.722375 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:22.721847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:55027)
0925 22:52:22.722180 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:22.722265 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260925 22:52:22.888821 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:22.888318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:37853)
0925 22:52:22.888656 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:22.888724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260925 22:52:23.076397 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:23.075917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:55309)
0925 22:52:23.076229 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:23.076297 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260925 22:52:23.723572 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:23.723109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:38917)
0925 22:52:23.723400 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:23.723462 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
I20260925 22:52:23.890157 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:23.889688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:50679)
0925 22:52:23.889993 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:23.890060 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260925 22:52:24.077801 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:24.077310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:56835)
0925 22:52:24.077614 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:24.077685 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260925 22:52:24.425433 20318 raft_consensus.cc:482] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:24.425633 20318 raft_consensus.cc:504] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "803bce081f4946018d306cfa88c2751e" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 45309 } } peers { permanent_uuid: "d7b35fc288894c20ab4f2793a8deb728" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 40925 } } peers { permanent_uuid: "f03679e2b39041099ebcf6bf8b05d180" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33529 } }
I20260925 22:52:24.426271 20318 leader_election.cc:288] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309), d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925)
W20260925 22:52:24.426915 19893 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b35fc288894c20ab4f2793a8deb728 (127.4.219.130:40925): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
W20260925 22:52:24.427012 19891 leader_election.cc:341] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 803bce081f4946018d306cfa88c2751e (127.4.219.131:45309): Not found: Tablet not found: 70cb14d326cb46fa88cc8db2724a8563
I20260925 22:52:24.427074 19891 leader_election.cc:302] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03679e2b39041099ebcf6bf8b05d180; no voters: 803bce081f4946018d306cfa88c2751e, d7b35fc288894c20ab4f2793a8deb728
I20260925 22:52:24.427209 20318 raft_consensus.cc:2726] T 70cb14d326cb46fa88cc8db2724a8563 P f03679e2b39041099ebcf6bf8b05d180 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:52:24.724807 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:24.724346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:45757)
0925 22:52:24.724658 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:24.724713 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":76,"threads_started":1}
I20260925 22:52:24.891294 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:24.890887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:58353)
0925 22:52:24.891140 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:24.891197 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260925 22:52:25.079033 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:25.078605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:51611)
0925 22:52:25.078880 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:25.078940 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260925 22:52:25.725978 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:25.725597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:47081)
0925 22:52:25.725831 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:25.725886 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260925 22:52:25.892485 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:25.892049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:60731)
0925 22:52:25.892334 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:25.892391 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260925 22:52:26.080171 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:26.079762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.131:48561)
0925 22:52:26.080004 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:26.080076 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260925 22:52:26.727375 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:26.726958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.129:36155)
0925 22:52:26.727215 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:26.727275 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260925 22:52:26.893829 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:26.893431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:39645 (local address 127.4.219.130:60633)
0925 22:52:26.893685 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:26.893735 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260925 22:52:27.066620 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 19840
I20260925 22:52:27.072379 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20007
I20260925 22:52:27.077956 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20142
2026-09-25T22:52:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19238 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:52:27 dist-test-slave-kb1g krb5kdc[20329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:52:27 dist-test-slave-kb1g krb5kdc[20329](info): set up 2 sockets
Sep 25 22:52:27 dist-test-slave-kb1g krb5kdc[20329](info): commencing operation
krb5kdc: starting...
W20260925 22:52:28.112803 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:52:28.232007 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:28.342578 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:28.342905 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:28.342980 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:28.347097 20342 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:52:28.347184 20342 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:52:28.347211 20342 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:52:28.347242 20342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:28.347285 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:28.347323 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:28.347383 20342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:28.347422 20342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:28.352843 20342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:34531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:28.353987 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:28.355062 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:28.363768 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:28.363821 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:28.363792 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:28.363946 20342 server_base.cc:1034] running on GCE node
I20260925 22:52:28.364643 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:28.365679 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:28.366864 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790376748366846 us; error 30 us; skew 500 ppm
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:28.372620 20342 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:28.373487 20342 webserver.cc:460] Webserver started at http://127.4.219.190:43735/ using document root <none> and password file <none>
I20260925 22:52:28.374001 20342 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:28.374058 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:28.374215 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:28.376149 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "45aa2142d58442c99c0016f3dc78d435"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "67ef7689ff6c14d8f377456cb76cd3b8"
server_key_iv: "70b4e5fe3c6e6904dd93286949dd9fa4"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.376716 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "45aa2142d58442c99c0016f3dc78d435"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "67ef7689ff6c14d8f377456cb76cd3b8"
server_key_iv: "70b4e5fe3c6e6904dd93286949dd9fa4"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.380189 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:52:28.382702 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.383976 20342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:28.384109 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "45aa2142d58442c99c0016f3dc78d435"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "67ef7689ff6c14d8f377456cb76cd3b8"
server_key_iv: "70b4e5fe3c6e6904dd93286949dd9fa4"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.384224 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:28.428085 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:28.430979 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:28.431198 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:28.438338 20342 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:34531
I20260925 22:52:28.438624 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:34531 every 8 connection(s)
I20260925 22:52:28.439420 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:28.442267 20410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:28.447584 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap starting.
I20260925 22:52:28.448388 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20342
I20260925 22:52:28.448482 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:52:28.448776 4974 external_mini_cluster.cc:1428] Setting key 4dc55ca3d5463ef2d95d6f469d46f992
I20260925 22:52:28.450356 20410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:28.451076 20410 log.cc:824] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:28.453184 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: No bootstrap required, opened a new log
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:28.457705 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:28.457952 20410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:28.458031 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Initialized, Role: FOLLOWER
I20260925 22:52:28.458492 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:28.458645 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:28.458737 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:28.458855 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:28.460093 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:28.460477 20410 leader_election.cc:302] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45aa2142d58442c99c0016f3dc78d435; no voters:
I20260925 22:52:28.460850 20410 leader_election.cc:288] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:52:28.460949 20415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:28.461185 20415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 LEADER]: Becoming Leader. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Running, Role: LEADER
I20260925 22:52:28.461541 20415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:28.462198 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:28.463361 20417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45aa2142d58442c99c0016f3dc78d435. Latest consensus state: current_term: 1 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:52:28.463474 20417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:28.463620 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:28.449885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.4.219.190:34531)
0925 22:52:28.450287 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:28.450301 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:28.450335 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:52:28.450989 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:28.451006 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:28.451372 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:28.451754 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:28.452627 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.453626 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.454762 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.455219 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.455332 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:28.458143 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:28.458151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:28.458173 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:28.458199 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:28.460270 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.460757 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.460765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.460773 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.460921 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.461282 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.461285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.461287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.461764 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:28.461958 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:28.462272 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 22:52:28.462575 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260925 22:52:28.463798 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:52:28.463876 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
W20260925 22:52:28.467470 20430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:52:28.467549 20430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:52:28.467621 20424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:28.468499 20424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:28.473796 20424 catalog_manager.cc:1347] Generated new cluster ID: 5c5caadd8a474ccfaa4f41165b0ad880
I20260925 22:52:28.473865 20424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:28.486372 20424 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:52:28.487609 20424 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:28.501001 20424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Generated new TSK 0
I20260925 22:52:28.501904 20424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:52:28.529573 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:28.639806 20436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:28.640098 20436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:28.640188 20436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:28.644151 20436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:28.644246 20436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:28.644338 20436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:28.649554 20436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:28.650638 20436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:28.651665 20436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:28.661201 20442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:28.661201 20441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:28.661257 20436 server_base.cc:1034] running on GCE node
W20260925 22:52:28.661201 20444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:28.661861 20436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:28.662505 20436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:28.663692 20436 hybrid_clock.cc:648] HybridClock initialized: now 1790376748663672 us; error 39 us; skew 500 ppm
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:28.666504 20436 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:28.667357 20436 webserver.cc:460] Webserver started at http://127.4.219.129:44575/ using document root <none> and password file <none>
I20260925 22:52:28.667900 20436 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:28.667989 20436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:28.668234 20436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:28.670240 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "9d6ff17d29af49afb1a13921064c6d38"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "106c88c40e3659eb74ddd053c41f6ea8"
server_key_iv: "c8051b5cead7595a614cce82fd5e87f5"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.670804 20436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "9d6ff17d29af49afb1a13921064c6d38"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "106c88c40e3659eb74ddd053c41f6ea8"
server_key_iv: "c8051b5cead7595a614cce82fd5e87f5"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.674438 20436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:52:28.676771 20451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.677963 20436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:28.678097 20436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "9d6ff17d29af49afb1a13921064c6d38"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "106c88c40e3659eb74ddd053c41f6ea8"
server_key_iv: "c8051b5cead7595a614cce82fd5e87f5"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.678208 20436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:28.690089 20436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:28.692739 20436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:28.692927 20436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:28.693466 20436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:28.694276 20436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:28.694346 20436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.694415 20436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:28.694469 20436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.703290 20436 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:38325
I20260925 22:52:28.703313 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:38325 every 8 connection(s)
I20260925 22:52:28.704142 20436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:28.705932 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20436
I20260925 22:52:28.706089 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:52:28.706382 4974 external_mini_cluster.cc:1428] Setting key 3a46a2ee241c73c15ef7fa79ee354482
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:52:28.718729 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:28.705897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52957 (local address 127.4.219.190:34531)
0925 22:52:28.706037 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:28.706043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:28.707335 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 22:52:28.708279 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:28.708282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:28.708332 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:28.708409 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:28.709971 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.710504 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.711674 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.711922 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.712050 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:28.714679 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:28.714683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:28.714686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:28.714718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:28.716437 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.716961 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.716967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.716972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.717060 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.717455 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.717459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.717460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.717656 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:28.717739 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:28.718328 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 22:52:28.718433 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:52:28.719496 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:28.706393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:52957)
0925 22:52:28.707189 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:28.707248 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:28.708089 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:28.708628 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:28.708636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:28.709100 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:28.709809 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:28.709829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.710624 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.710627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:28.711549 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:28.711558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.712086 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.712090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:28.712249 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:28.712895 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:28.712910 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:28.714509 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:28.716582 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:28.716594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:28.716607 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:28.716853 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:28.717141 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:28.717143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:28.717145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:28.717301 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:28.717751 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:28.717757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:28.718076 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 22:52:28.718238 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 22:52:28.718573 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":144,"threads_started":1}
I20260925 22:52:28.721058 20565 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:28.721422 20565 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:28.721940 20565 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:28.723476 20374 ts_manager.cc:194] Registered new tserver with Master: 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:28.724702 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:52957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:52:28.735260 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:28.844900 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:28.845156 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:28.845199 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:28.849164 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:28.849226 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:28.849284 20570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:28.854523 20570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:28.855549 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:28.856521 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:28.866410 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:28.866411 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:28.866393 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:28.866585 20570 server_base.cc:1034] running on GCE node
I20260925 22:52:28.867096 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:28.867755 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:28.868916 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790376748868890 us; error 36 us; skew 500 ppm
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:28.871564 20570 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:28.872423 20570 webserver.cc:460] Webserver started at http://127.4.219.130:37833/ using document root <none> and password file <none>
I20260925 22:52:28.873010 20570 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:28.873101 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:28.873328 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:28.875303 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "f3fce4008f5e47dfb0a109d64063b97c"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "ad836e7769dd118eb5279a857b4e7c02"
server_key_iv: "0e03a7a480d51bcd9fa7e4490b60cb51"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.875880 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "f3fce4008f5e47dfb0a109d64063b97c"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "ad836e7769dd118eb5279a857b4e7c02"
server_key_iv: "0e03a7a480d51bcd9fa7e4490b60cb51"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.879587 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:52:28.882385 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.883682 20570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:52:28.883827 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "f3fce4008f5e47dfb0a109d64063b97c"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "ad836e7769dd118eb5279a857b4e7c02"
server_key_iv: "0e03a7a480d51bcd9fa7e4490b60cb51"
server_key_version: "encryptionkey@0"
I20260925 22:52:28.883952 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:28.894446 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:28.897464 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:28.897694 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:28.898284 20570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:28.899124 20570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:28.899194 20570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.899266 20570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:28.899312 20570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:28.908452 20570 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:38155
I20260925 22:52:28.908473 20698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:38155 every 8 connection(s)
I20260925 22:52:28.909361 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:28.911526 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20570
I20260925 22:52:28.911681 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:52:28.911947 4974 external_mini_cluster.cc:1428] Setting key 87a9445d43f73ba49f0db0af51645628
Sep 25 22:52:28 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376748, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:52:28.923871 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:28.911615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:39997 (local address 127.4.219.190:34531)
0925 22:52:28.911757 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:28.911762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:28.912911 (+ 1149us) server_negotiation.cc:407] Connection header received
0925 22:52:28.914109 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:28.914111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:28.914152 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:28.914212 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:28.915735 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.916270 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.917565 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.917806 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.917917 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:28.920398 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:28.920405 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:28.920410 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:28.920446 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:28.921894 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.922446 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.922448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.922449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.922525 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:28.922807 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:28.922809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:28.922810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:28.922974 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:28.923068 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:28.923510 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 22:52:28.923638 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:52:28.924742 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:28.911917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:39997)
0925 22:52:28.912752 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:28.912806 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:28.913861 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:28.914417 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:28.914428 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:28.914967 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:28.915594 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:28.915609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.916398 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.916402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:28.917439 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:28.917458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:28.917964 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:28.917968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:28.918148 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:28.918777 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:28.918793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:28.920220 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:28.922025 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:28.922032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:28.922051 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:28.922360 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:28.922603 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:28.922605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:28.922607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:28.922730 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:28.923083 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:28.923088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:28.923308 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 22:52:28.923463 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 22:52:28.923798 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":131,"threads_started":1}
I20260925 22:52:28.925981 20699 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:28.926244 20699 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:28.926818 20699 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:28.927915 20374 ts_manager.cc:194] Registered new tserver with Master: f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
I20260925 22:52:28.928818 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:39997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:52:28.939266 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:29.049521 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:29.049784 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:29.049826 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:29.053857 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:29.053944 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:29.054040 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:29.060487 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:29.061529 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:29.062559 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:29.071915 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:29.071911 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:29.071955 20704 server_base.cc:1034] running on GCE node
W20260925 22:52:29.071914 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:29.072643 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:29.073271 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:29.074425 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790376749074409 us; error 26 us; skew 500 ppm
Sep 25 22:52:29 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376749, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:29.077031 20704 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:29.077832 20704 webserver.cc:460] Webserver started at http://127.4.219.131:46629/ using document root <none> and password file <none>
I20260925 22:52:29.078367 20704 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:29.078457 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:29.078692 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:52:29.080747 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "58d5985b20224a43823526246b334235"
format_stamp: "Formatted at 2026-09-25 22:52:29 on dist-test-slave-kb1g"
server_key: "b5f05fc66589ecb648152bc5f99ee21c"
server_key_iv: "26bd213516a1f61ab0d9918694e738b0"
server_key_version: "encryptionkey@0"
I20260925 22:52:29.081346 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "58d5985b20224a43823526246b334235"
format_stamp: "Formatted at 2026-09-25 22:52:29 on dist-test-slave-kb1g"
server_key: "b5f05fc66589ecb648152bc5f99ee21c"
server_key_iv: "26bd213516a1f61ab0d9918694e738b0"
server_key_version: "encryptionkey@0"
I20260925 22:52:29.085036 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:52:29.087375 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:29.088871 20704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:29.088960 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "58d5985b20224a43823526246b334235"
format_stamp: "Formatted at 2026-09-25 22:52:29 on dist-test-slave-kb1g"
server_key: "b5f05fc66589ecb648152bc5f99ee21c"
server_key_iv: "26bd213516a1f61ab0d9918694e738b0"
server_key_version: "encryptionkey@0"
I20260925 22:52:29.089056 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:29.106822 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:29.110213 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:29.110369 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:29.110891 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:29.111663 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:29.111709 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:29.111748 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:29.111766 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:29.120077 20704 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:39539
I20260925 22:52:29.120141 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:39539 every 8 connection(s)
I20260925 22:52:29.121066 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:29.125070 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20704
I20260925 22:52:29.125207 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:52:29.125465 4974 external_mini_cluster.cc:1428] Setting key 9fda75ec4fa3c69c623f01efd3b4c836
Sep 25 22:52:29 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376749, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:29.134086 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.122859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:34977 (local address 127.4.219.190:34531)
0925 22:52:29.122999 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.123004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.124255 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 22:52:29.125181 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.125185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.125239 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.125351 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:29.126765 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.127365 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.128473 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.128807 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.128888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.131149 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:29.131151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:29.131153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:29.131177 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:29.132432 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:29.132878 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:29.132881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:29.132882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:29.132953 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:29.133231 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:29.133233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:29.133234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:29.133390 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:29.133462 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.133883 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.133973 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:52:29.134881 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.123245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:34977)
0925 22:52:29.124104 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.124145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.124972 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.125528 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.125535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.126035 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:29.126615 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.126635 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.127490 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.127494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.128374 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.128382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.128931 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.128934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.129084 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.129600 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:29.129615 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:29.130993 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:29.132545 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:29.132550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:29.132561 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:29.132794 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:29.133035 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:29.133037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:29.133038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:29.133153 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:29.133470 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:29.133475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:29.133673 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.133817 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.134055 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":531,"threads_started":1}
I20260925 22:52:29.135928 20833 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:29.136165 20833 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:29.136683 20833 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:29.137643 20374 ts_manager.cc:194] Registered new tserver with Master: 58d5985b20224a43823526246b334235 (127.4.219.131:39539)
I20260925 22:52:29.138352 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:34977
I20260925 22:52:29.139256 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:52:29.147593 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.141213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.4.219.190:34531)
0925 22:52:29.141320 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.141323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.141501 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:52:29.141595 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.141597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.141629 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.141692 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:29.142291 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.142767 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.143513 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.143707 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.143778 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.145096 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:29.145099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:29.145101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:29.145118 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:29.146272 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:29.146633 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:29.146636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:29.146637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:29.146702 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:29.146918 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:29.146920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:29.146921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:29.147102 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:29.147199 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.147397 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.147480 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:52:29.150143 20374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:52:29.153100 20374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:52:29.167234 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.0.0.1:38762)
0925 22:52:29.162478 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.162508 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.162658 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.163339 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.163342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.163359 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.163426 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.163432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164839 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.164842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.165950 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.165960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.166783 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.166880 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.166903 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.166985 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.167086 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":85,"thread_start_us":89,"threads_started":1}
I20260925 22:52:29.167253 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.4.219.129:38325)
0925 22:52:29.162334 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.162346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.162586 (+ 240us) server_negotiation.cc:407] Connection header received
0925 22:52:29.162784 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.162788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.162962 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.163176 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.163559 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164676 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166086 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.166661 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166765 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.166903 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.167015 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.167112 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":138,"threads_started":1}
I20260925 22:52:29.167279 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.4.219.131:39539)
0925 22:52:29.162264 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.162279 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.162578 (+ 299us) server_negotiation.cc:407] Connection header received
0925 22:52:29.162652 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.162662 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.162825 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.162967 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.163267 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164109 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.165941 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.166543 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166625 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.166759 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.167077 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.167158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":131,"threads_started":1}
I20260925 22:52:29.168103 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.0.0.1:35754)
0925 22:52:29.161942 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.162021 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.162154 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.162940 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.162946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.162967 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.163150 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.163157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164579 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.164583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.165814 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.165825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.167739 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.167742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.167807 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.167824 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.167874 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.167960 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":89,"thread_start_us":70,"threads_started":1}
I20260925 22:52:29.168190 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.4.219.130:38155)
0925 22:52:29.161974 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.162001 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.162291 (+ 290us) server_negotiation.cc:407] Connection header received
0925 22:52:29.162377 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.162383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.162607 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.162746 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.163283 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164430 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166811 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.167627 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.167737 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.167888 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.167937 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.168038 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":110,"threads_started":1}
I20260925 22:52:29.167928 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.161249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:39539 (local address 127.0.0.1:48248)
0925 22:52:29.162403 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.162415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.162495 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.162958 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.162961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.162979 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.163134 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.163143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.164233 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.164237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.165814 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.165825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.166663 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.166667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.166780 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.166816 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.166922 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.167798 (+ 876us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":817,"thread_start_us":101,"threads_started":1}
I20260925 22:52:29.171025 20767 tablet_service.cc:1467] Processing CreateTablet for tablet a6927b1af9134fabbcbfffc66f361d94 (DEFAULT_TABLE table=test-table [id=d258bcd94fec48df8e35682d0a8ba275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:29.171051 20633 tablet_service.cc:1467] Processing CreateTablet for tablet a6927b1af9134fabbcbfffc66f361d94 (DEFAULT_TABLE table=test-table [id=d258bcd94fec48df8e35682d0a8ba275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:29.171051 20499 tablet_service.cc:1467] Processing CreateTablet for tablet a6927b1af9134fabbcbfffc66f361d94 (DEFAULT_TABLE table=test-table [id=d258bcd94fec48df8e35682d0a8ba275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:29.172034 20499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6927b1af9134fabbcbfffc66f361d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:29.172034 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6927b1af9134fabbcbfffc66f361d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:29.172053 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6927b1af9134fabbcbfffc66f361d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:29.178109 20850 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Bootstrap starting.
I20260925 22:52:29.179739 20851 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap starting.
I20260925 22:52:29.180841 20852 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Bootstrap starting.
I20260925 22:52:29.180965 20850 tablet_bootstrap.cc:654] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:29.181838 20850 log.cc:824] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:29.181985 20851 tablet_bootstrap.cc:654] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:29.183171 20851 log.cc:824] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:29.183300 20852 tablet_bootstrap.cc:654] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:29.184091 20850 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: No bootstrap required, opened a new log
I20260925 22:52:29.184062 20852 log.cc:824] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:29.184278 20850 ts_tablet_manager.cc:1397] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:52:29.185164 20851 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: No bootstrap required, opened a new log
I20260925 22:52:29.185357 20851 ts_tablet_manager.cc:1397] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 22:52:29.186053 20852 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: No bootstrap required, opened a new log
I20260925 22:52:29.186232 20852 ts_tablet_manager.cc:1397] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:52:29.188315 20850 raft_consensus.cc:348] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.188668 20850 raft_consensus.cc:374] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:29.188513 20851 raft_consensus.cc:348] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.188752 20850 raft_consensus.cc:729] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d5985b20224a43823526246b334235, State: Initialized, Role: FOLLOWER
I20260925 22:52:29.188796 20851 raft_consensus.cc:374] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:29.188858 20851 raft_consensus.cc:729] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6ff17d29af49afb1a13921064c6d38, State: Initialized, Role: FOLLOWER
I20260925 22:52:29.189211 20850 consensus_queue.cc:260] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.189276 20851 consensus_queue.cc:260] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.189925 20852 raft_consensus.cc:348] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.190181 20852 raft_consensus.cc:374] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:29.190272 20852 raft_consensus.cc:729] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fce4008f5e47dfb0a109d64063b97c, State: Initialized, Role: FOLLOWER
I20260925 22:52:29.190287 20850 ts_tablet_manager.cc:1428] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:52:29.190294 20851 ts_tablet_manager.cc:1428] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260925 22:52:29.190625 20565 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
I20260925 22:52:29.190629 20833 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
I20260925 22:52:29.190655 20852 consensus_queue.cc:260] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.191250 20699 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
I20260925 22:52:29.191529 20852 ts_tablet_manager.cc:1428] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260925 22:52:29.205253 20566 tablet.cc:2369] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:29.281069 20856 raft_consensus.cc:482] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:29.281277 20856 raft_consensus.cc:504] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.282528 20856 leader_election.cc:288] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 58d5985b20224a43823526246b334235 (127.4.219.131:39539)
I20260925 22:52:29.283653 20857 raft_consensus.cc:482] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:29.283825 20857 raft_consensus.cc:504] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.285080 20857 leader_election.cc:288] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:29.287391 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.282971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:32821 (local address 127.4.219.130:38155)
0925 22:52:29.283116 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.283120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.283458 (+ 338us) server_negotiation.cc:407] Connection header received
0925 22:52:29.283529 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.283532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.283585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.283672 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.283978 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.284712 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286065 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.286682 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286778 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.286859 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.287182 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.287259 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260925 22:52:29.287390 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.282918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:52453 (local address 127.4.219.131:39539)
0925 22:52:29.283052 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.283057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.283217 (+ 160us) server_negotiation.cc:407] Connection header received
0925 22:52:29.283313 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.283318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.283384 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.283460 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.283851 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.284424 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286008 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.286682 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286784 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.286828 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.287200 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.287264 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":50}
I20260925 22:52:29.287847 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.282755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:39539 (local address 127.4.219.129:52453)
0925 22:52:29.283121 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.283134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.283227 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.283541 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.283543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.283558 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.283708 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.283715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.284590 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.284593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.285848 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.285859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.286891 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.287017 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.287044 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.287566 (+ 522us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.287704 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
I20260925 22:52:29.288152 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.282755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.4.219.129:32821)
0925 22:52:29.283313 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.283326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.283435 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.283690 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.283693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.283709 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.283838 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.283844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.284820 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.284824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.285936 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.285947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.286959 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.286962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.287050 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.287069 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.287797 (+ 728us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.287870 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":106,"thread_start_us":118,"threads_started":1}
I20260925 22:52:29.288595 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "9d6ff17d29af49afb1a13921064c6d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d5985b20224a43823526246b334235" is_pre_election: true
I20260925 22:52:29.288856 20787 raft_consensus.cc:2454] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6ff17d29af49afb1a13921064c6d38 in term 0.
I20260925 22:52:29.289381 20455 leader_election.cc:302] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5985b20224a43823526246b334235, 9d6ff17d29af49afb1a13921064c6d38; no voters:
I20260925 22:52:29.289479 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "9d6ff17d29af49afb1a13921064c6d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c" is_pre_election: true
I20260925 22:52:29.289703 20856 raft_consensus.cc:2781] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:29.289746 20653 raft_consensus.cc:2454] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6ff17d29af49afb1a13921064c6d38 in term 0.
I20260925 22:52:29.289778 20856 raft_consensus.cc:482] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:29.289821 20856 raft_consensus.cc:3037] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:29.290453 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.285189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.4.219.131:56583)
0925 22:52:29.285345 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.285360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.285436 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.286437 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.286443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.286458 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.286564 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.286572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.287678 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.287681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.289170 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.289177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.290027 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.290029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.290172 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.290204 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.290268 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.290342 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":272}
I20260925 22:52:29.290474 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.285914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:54927 (local address 127.4.219.129:38325)
0925 22:52:29.286028 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.286032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.286043 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:52:29.286095 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.286099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.286143 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.286214 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.286614 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.287397 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.289170 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.289766 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.289853 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.289898 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.290316 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.290383 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:52:29.290827 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.285339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.4.219.131:54927)
0925 22:52:29.285821 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.285835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:29.285922 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:29.286313 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:29.286316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:29.286334 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:29.286477 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.286486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.287613 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.287618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.289040 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:29.289054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.289994 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.289999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:29.290103 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.290203 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:52:29.290629 (+ 426us) client_negotiation.cc:240] Negotiation successful
0925 22:52:29.290701 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":53,"thread_start_us":115,"threads_started":1}
I20260925 22:52:29.291131 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.285657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:56583 (local address 127.4.219.130:38155)
0925 22:52:29.286217 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.286225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.286236 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:52:29.286279 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.286284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.286325 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.286663 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:29.286754 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.287564 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.289283 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.289898 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.290717 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.290763 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.290797 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.290868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":103,"threads_started":1}
I20260925 22:52:29.291198 20856 raft_consensus.cc:504] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.291311 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6ff17d29af49afb1a13921064c6d38" is_pre_election: true
I20260925 22:52:29.291482 20519 raft_consensus.cc:2379] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58d5985b20224a43823526246b334235 in current term 1: Already voted for candidate 9d6ff17d29af49afb1a13921064c6d38 in this term.
I20260925 22:52:29.291435 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c" is_pre_election: true
I20260925 22:52:29.291574 20856 leader_election.cc:288] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 1 election: Requested vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 58d5985b20224a43823526246b334235 (127.4.219.131:39539)
I20260925 22:52:29.291590 20653 raft_consensus.cc:2454] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d5985b20224a43823526246b334235 in term 0.
I20260925 22:52:29.291896 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "9d6ff17d29af49afb1a13921064c6d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c"
I20260925 22:52:29.292014 20653 raft_consensus.cc:3037] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:29.292076 20722 leader_election.cc:302] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c; no voters:
I20260925 22:52:29.292328 20857 raft_consensus.cc:2781] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:29.292435 20857 raft_consensus.cc:482] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:29.292506 20857 raft_consensus.cc:3037] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:29.292704 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "9d6ff17d29af49afb1a13921064c6d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58d5985b20224a43823526246b334235"
I20260925 22:52:29.293247 20653 raft_consensus.cc:2454] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6ff17d29af49afb1a13921064c6d38 in term 1.
I20260925 22:52:29.293514 20454 leader_election.cc:302] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38, f3fce4008f5e47dfb0a109d64063b97c; no voters:
I20260925 22:52:29.293701 20857 raft_consensus.cc:504] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.293745 20856 raft_consensus.cc:2781] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:29.293918 20787 raft_consensus.cc:2379] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d6ff17d29af49afb1a13921064c6d38 in current term 1: Already voted for candidate 58d5985b20224a43823526246b334235 in this term.
I20260925 22:52:29.294037 20856 raft_consensus.cc:686] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6ff17d29af49afb1a13921064c6d38, State: Running, Role: LEADER
I20260925 22:52:29.294065 20857 leader_election.cc:288] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 election: Requested vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:29.294389 20856 consensus_queue.cc:237] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:29.294543 20653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c"
I20260925 22:52:29.294561 20519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6927b1af9134fabbcbfffc66f361d94" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6ff17d29af49afb1a13921064c6d38"
I20260925 22:52:29.294675 20653 raft_consensus.cc:2379] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 58d5985b20224a43823526246b334235 in current term 1: Already voted for candidate 9d6ff17d29af49afb1a13921064c6d38 in this term.
I20260925 22:52:29.294974 20722 leader_election.cc:302] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58d5985b20224a43823526246b334235; no voters: 9d6ff17d29af49afb1a13921064c6d38, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:52:29.295115 20857 raft_consensus.cc:2726] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 22:52:29.297468 20373 catalog_manager.cc:5495] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "9d6ff17d29af49afb1a13921064c6d38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } health_report { overall_health: HEALTHY } } }
I20260925 22:52:29.318197 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.314591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.4.219.129:38325)
0925 22:52:29.314739 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:29.314743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:29.314759 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:52:29.314904 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:29.314907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:29.314960 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:29.315074 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:29.315312 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.315816 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.316749 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:29.316970 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:29.317080 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:29.317394 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:29.317819 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:29.317957 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:29.317995 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 22:52:29.318050 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:52:29.323976 20653 raft_consensus.cc:1261] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Refusing update from remote peer 9d6ff17d29af49afb1a13921064c6d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:29.323976 20787 raft_consensus.cc:1261] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Refusing update from remote peer 9d6ff17d29af49afb1a13921064c6d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:29.324630 20856 consensus_queue.cc:1035] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:29.324828 20863 consensus_queue.cc:1035] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:29.332661 20868 mvcc.cc:205] Tried to move back new op lower bound from 7333383165224067072 to 7333383165112934400. Current Snapshot: MvccSnapshot[applied={T|T < 7333383165224067072}]
I20260925 22:52:29.333343 20867 mvcc.cc:205] Tried to move back new op lower bound from 7333383165224067072 to 7333383165112934400. Current Snapshot: MvccSnapshot[applied={T|T < 7333383165224067072}]
I20260925 22:52:29.341747 20374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:29.341933 20374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:29.344192 20374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Sending DeleteTablet for 3 replicas of tablet a6927b1af9134fabbcbfffc66f361d94
I20260925 22:52:29.344980 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet a6927b1af9134fabbcbfffc66f361d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:29 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:35754
I20260925 22:52:29.344985 20499 tablet_service.cc:1514] Processing DeleteTablet for tablet a6927b1af9134fabbcbfffc66f361d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:29 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:38762
I20260925 22:52:29.345474 20877 tablet_replica.cc:330] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: stopping tablet replica
I20260925 22:52:29.345734 20876 tablet_replica.cc:330] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: stopping tablet replica
I20260925 22:52:29.345675 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet a6927b1af9134fabbcbfffc66f361d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:29 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:48248
I20260925 22:52:29.345860 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20436
I20260925 22:52:29.346145 20876 raft_consensus.cc:2229] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:29.346447 20876 raft_consensus.cc:2258] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:29.346554 20878 tablet_replica.cc:330] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: stopping tablet replica
I20260925 22:52:29.346837 20878 raft_consensus.cc:2229] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:29.347100 20878 raft_consensus.cc:2258] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:29.348417 20878 ts_tablet_manager.cc:1905] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:29.348417 20876 ts_tablet_manager.cc:1905] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:29.350584 20878 ts_tablet_manager.cc:1918] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:29.350695 20878 log.cc:1196] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/a6927b1af9134fabbcbfffc66f361d94
I20260925 22:52:29.351001 20878 ts_tablet_manager.cc:1939] T a6927b1af9134fabbcbfffc66f361d94 P 58d5985b20224a43823526246b334235: Deleting consensus metadata
I20260925 22:52:29.352150 20361 catalog_manager.cc:4841] TS 58d5985b20224a43823526246b334235 (127.4.219.131:39539): tablet a6927b1af9134fabbcbfffc66f361d94 (table test-table [id=d258bcd94fec48df8e35682d0a8ba275]) successfully deleted
I20260925 22:52:29.352280 20876 ts_tablet_manager.cc:1918] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:29.352391 20876 log.cc:1196] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/a6927b1af9134fabbcbfffc66f361d94
I20260925 22:52:29.352741 20876 ts_tablet_manager.cc:1939] T a6927b1af9134fabbcbfffc66f361d94 P f3fce4008f5e47dfb0a109d64063b97c: Deleting consensus metadata
I20260925 22:52:29.354069 20360 catalog_manager.cc:4841] TS f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): tablet a6927b1af9134fabbcbfffc66f361d94 (table test-table [id=d258bcd94fec48df8e35682d0a8ba275]) successfully deleted
W20260925 22:52:29.354166 20358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:38325 (error 108)
I20260925 22:52:29.354903 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20570
I20260925 22:52:29.355238 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:29.354889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.0.0.1:38770)
0925 22:52:29.355054 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:29.355135 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 22:52:29.356070 20358 catalog_manager.cc:4584] TS 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6927b1af9134fabbcbfffc66f361d94: Network error: Client connection negotiation failed: client connection to 127.4.219.129:38325: connect: Connection refused (error 111)
I20260925 22:52:29.360957 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20704
I20260925 22:52:29.366602 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20342
I20260925 22:52:49.373217 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--webserver_interface=127.4.219.190
--webserver_port=43735
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:52:49.492707 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:49.493010 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:49.493091 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:49.497109 20880 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:52:49.497193 20880 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:52:49.497249 20880 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:52:49.497273 20880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:52:49.497316 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:49.497354 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:49.497388 20880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:52:49.497422 20880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:52:49.502964 20880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:34531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=43735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:49.504253 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:49.505399 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:49.518800 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.518800 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.519021 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:49.518992 20880 server_base.cc:1034] running on GCE node
I20260925 22:52:49.519922 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:49.521180 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:49.522339 20880 hybrid_clock.cc:648] HybridClock initialized: now 1790376769522325 us; error 30 us; skew 500 ppm
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:49.525193 20880 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:52:49.526121 20880 webserver.cc:460] Webserver started at http://127.4.219.190:43735/ using document root <none> and password file <none>
I20260925 22:52:49.526641 20880 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:49.526710 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:49.530604 20880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:52:49.532830 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:49.534092 20880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:49.534233 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "45aa2142d58442c99c0016f3dc78d435"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "67ef7689ff6c14d8f377456cb76cd3b8"
server_key_iv: "70b4e5fe3c6e6904dd93286949dd9fa4"
server_key_version: "encryptionkey@0"
I20260925 22:52:49.534835 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:49.545922 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:49.549091 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:49.549312 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:49.556365 20880 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:34531
I20260925 22:52:49.556675 20948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:34531 every 8 connection(s)
I20260925 22:52:49.557528 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:52:49.559700 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20880
I20260925 22:52:49.560179 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:38325
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=44575
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:49.562434 20949 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:52:49.564178 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap starting.
I20260925 22:52:49.581132 20949 log.cc:824] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:49.589597 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:49.590006 20949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap complete.
I20260925 22:52:49.594208 20949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:49.594640 20949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Initialized, Role: FOLLOWER
I20260925 22:52:49.595116 20949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:49.595225 20949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:52:49.595316 20949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:52:49.595429 20949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:52:49.597409 20949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:49.597745 20949 leader_election.cc:302] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45aa2142d58442c99c0016f3dc78d435; no voters:
I20260925 22:52:49.598039 20949 leader_election.cc:288] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:52:49.598181 20953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:52:49.598420 20953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 LEADER]: Becoming Leader. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Running, Role: LEADER
I20260925 22:52:49.598795 20953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:52:49.599300 20949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:52:49.600212 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:52:49.600327 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:49.601271 20962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:52:49.601787 20955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45aa2142d58442c99c0016f3dc78d435. Latest consensus state: current_term: 2 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:52:49.601898 20955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:52:49.606302 20962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:52:49.606879 20962 catalog_manager.cc:1259] Loaded cluster ID: 5c5caadd8a474ccfaa4f41165b0ad880
I20260925 22:52:49.606936 20962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:52:49.609297 20962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:52:49.609946 20962 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Loaded TSK: 0
I20260925 22:52:49.629640 20962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Generated new TSK 1
I20260925 22:52:49.629933 20962 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:52:49.679888 20951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:49.680152 20951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:49.680203 20951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:49.684245 20951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:49.684302 20951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:49.684396 20951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:52:49.689925 20951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:38325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=44575
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:49.690938 20951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:49.692039 20951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:49.701511 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.701469 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.701469 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:49.702286 20951 server_base.cc:1034] running on GCE node
I20260925 22:52:49.702670 20951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:49.703290 20951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:49.704474 20951 hybrid_clock.cc:648] HybridClock initialized: now 1790376769704450 us; error 43 us; skew 500 ppm
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:49.707167 20951 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:52:49.708061 20951 webserver.cc:460] Webserver started at http://127.4.219.129:44575/ using document root <none> and password file <none>
I20260925 22:52:49.708643 20951 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:49.708717 20951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:49.712471 20951 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:52:49.714946 20986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:49.716359 20951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:52:49.716497 20951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "9d6ff17d29af49afb1a13921064c6d38"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "106c88c40e3659eb74ddd053c41f6ea8"
server_key_iv: "c8051b5cead7595a614cce82fd5e87f5"
server_key_version: "encryptionkey@0"
I20260925 22:52:49.717121 20951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:49.738869 20951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:49.742308 20951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:49.742522 20951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:49.743177 20951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:49.744333 20993 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:52:49.748407 20951 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:52:49.748471 20951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 22:52:49.748540 20951 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:52:49.750222 20951 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:52:49.750277 20951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:52:49.750528 20993 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap starting.
I20260925 22:52:49.761065 20951 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:38325
I20260925 22:52:49.761111 21100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:38325 every 8 connection(s)
I20260925 22:52:49.762323 20951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:52:49.767087 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 20951
I20260925 22:52:49.767584 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:38155
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=37833
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:52:49.771951 20993 log.cc:824] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:49.778995 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.764468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:51769 (local address 127.4.219.190:34531)
0925 22:52:49.764867 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:49.764886 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:49.765223 (+ 337us) server_negotiation.cc:407] Connection header received
0925 22:52:49.766303 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:49.766312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:49.766592 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:49.766924 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:49.768213 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.769354 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.771009 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.771506 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.771697 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.774217 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:49.774222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:49.774234 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:49.774259 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:49.775907 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.776427 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.776433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.776439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.776602 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.776930 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.776935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.776937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.777274 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:49.777413 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:49.777924 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 22:52:49.778176 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":158,"threads_started":1}
I20260925 22:52:49.779006 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.764498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:51769)
0925 22:52:49.765052 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:49.765093 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:49.765895 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:49.766997 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:49.767005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:49.767460 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:49.768032 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.768047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.769539 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.769546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.770888 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.770898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.771649 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.771653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.771838 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.772472 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:49.772491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:49.774022 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:49.776042 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.776048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.776060 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.776314 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.776695 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.776699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.776701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.776836 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.777387 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:49.777393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:49.777631 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 22:52:49.777817 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 22:52:49.778132 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":127,"threads_started":1}
I20260925 22:52:49.781491 21101 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:49.781499 20993 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:52:49.781733 21101 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:49.781900 20993 tablet_bootstrap.cc:492] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap complete.
I20260925 22:52:49.782238 21101 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:49.783345 20993 ts_tablet_manager.cc:1397] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260925 22:52:49.784482 20913 ts_manager.cc:194] Registered new tserver with Master: 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:49.786192 20993 raft_consensus.cc:348] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:49.786691 20993 raft_consensus.cc:729] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6ff17d29af49afb1a13921064c6d38, State: Initialized, Role: FOLLOWER
I20260925 22:52:49.787201 20993 consensus_queue.cc:260] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:49.788269 20993 ts_tablet_manager.cc:1428] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:52:49.788640 20913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:51769
I20260925 22:52:49.791071 21101 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.129@KRBTEST.COM
I20260925 22:52:49.797835 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.787067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.0.0.1:49356)
0925 22:52:49.787727 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:49.787769 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:49.787888 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:49.788512 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:49.788517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:49.788930 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:49.789040 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.789047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.790447 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.790451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.791342 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.791352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.792095 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.792100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.792215 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.792826 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:49.792843 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:49.794332 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:49.796270 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.796278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.796281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.796488 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.796836 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.796838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.796840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.796913 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.797382 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:49.797386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:49.797475 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 22:52:49.797572 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:52:49.797716 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":105,"threads_started":1}
I20260925 22:52:49.797993 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.787159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.4.219.129:38325)
0925 22:52:49.787497 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:49.787508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:49.788010 (+ 502us) server_negotiation.cc:407] Connection header received
0925 22:52:49.788074 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:49.788079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:49.788266 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:49.788374 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:49.789161 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.790324 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.791456 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.791971 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.792082 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.794469 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:49.794478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:49.794490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:49.794515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:49.796153 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.796647 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.796653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.796655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.796747 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.797005 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.797008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.797010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.797281 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:49.797405 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:49.797686 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 22:52:49.797840 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260925 22:52:49.798933 21035 tablet_service.cc:1514] Processing DeleteTablet for tablet a6927b1af9134fabbcbfffc66f361d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:29 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:49356
I20260925 22:52:49.799780 21110 tablet_replica.cc:330] stopping tablet replica
I20260925 22:52:49.800020 21110 raft_consensus.cc:2229] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:49.800209 21110 raft_consensus.cc:2258] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:49.801182 21110 ts_tablet_manager.cc:1905] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:49.804715 21110 ts_tablet_manager.cc:1918] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:49.804791 21110 log.cc:1196] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal/wals/a6927b1af9134fabbcbfffc66f361d94
I20260925 22:52:49.805048 21110 ts_tablet_manager.cc:1939] T a6927b1af9134fabbcbfffc66f361d94 P 9d6ff17d29af49afb1a13921064c6d38: Deleting consensus metadata
I20260925 22:52:49.806571 20897 catalog_manager.cc:4841] TS 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325): tablet a6927b1af9134fabbcbfffc66f361d94 (table test-table [id=d258bcd94fec48df8e35682d0a8ba275]) successfully deleted
W20260925 22:52:49.887879 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:49.888203 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:49.888278 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:49.892330 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:49.892416 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:49.892508 21105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:52:49.897945 21105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:38155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=37833
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:49.898988 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:49.900089 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:49.909456 21118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.909457 21116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:49.909456 21115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:49.909749 21105 server_base.cc:1034] running on GCE node
I20260925 22:52:49.910146 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:49.910784 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:49.911966 21105 hybrid_clock.cc:648] HybridClock initialized: now 1790376769911952 us; error 29 us; skew 500 ppm
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:49.914713 21105 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:52:49.915520 21105 webserver.cc:460] Webserver started at http://127.4.219.130:37833/ using document root <none> and password file <none>
I20260925 22:52:49.916069 21105 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:49.916174 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:49.919862 21105 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:49.921919 21125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:49.923095 21105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:52:49.923224 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "f3fce4008f5e47dfb0a109d64063b97c"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "ad836e7769dd118eb5279a857b4e7c02"
server_key_iv: "0e03a7a480d51bcd9fa7e4490b60cb51"
server_key_version: "encryptionkey@0"
I20260925 22:52:49.923756 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:49.957418 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:49.960373 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:49.960602 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:49.961216 21105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:49.962031 21105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:49.962102 21105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:49.962172 21105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:49.962224 21105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:49.971014 21105 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:38155
I20260925 22:52:49.971038 21238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:38155 every 8 connection(s)
I20260925 22:52:49.971987 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:52:49.975481 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21105
I20260925 22:52:49.975983 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:39539
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=46629
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:52:49 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376769, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:49.986258 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.973777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:58887 (local address 127.4.219.190:34531)
0925 22:52:49.973922 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:49.973925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:49.974937 (+ 1012us) server_negotiation.cc:407] Connection header received
0925 22:52:49.975815 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:49.975820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:49.975883 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:49.975984 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:49.977308 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.977845 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.979155 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.979397 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.979522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.982233 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:49.982239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:49.982246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:49.982281 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:49.984035 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.984507 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.984512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.984516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.984633 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:49.984955 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:49.984960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:49.984965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:49.985255 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:49.985373 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:49.985840 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 22:52:49.986004 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:52:49.986927 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:49.974174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:58887)
0925 22:52:49.974786 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:49.974825 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:49.975561 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:49.976205 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:49.976219 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:49.976627 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:49.977112 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.977136 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.977969 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.977974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.979031 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:49.979040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:49.979569 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:49.979573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:49.979736 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:49.980347 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:49.980363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:49.982047 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:49.984154 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.984158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.984171 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.984407 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.984728 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:49.984730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:49.984731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:49.984857 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:49.985361 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:49.985366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:49.985579 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 22:52:49.985753 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:52:49.986007 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":98,"threads_started":1}
I20260925 22:52:49.988164 21239 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:49.988394 21239 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:49.988929 21239 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:49.989984 20913 ts_manager.cc:194] Registered new tserver with Master: f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
I20260925 22:52:49.990571 20913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:58887
W20260925 22:52:50.090437 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:52:50.090718 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:52:50.090760 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:52:50.094914 21242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:52:50.094977 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:52:50.095077 21242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:52:50.100745 21242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:39539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=46629
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:52:50.101877 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:52:50.102890 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:52:50.112134 21250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:52:50.112139 21248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:50.112192 21242 server_base.cc:1034] running on GCE node
W20260925 22:52:50.112169 21247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:52:50.112815 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:52:50.113472 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:52:50.114658 21242 hybrid_clock.cc:648] HybridClock initialized: now 1790376770114640 us; error 33 us; skew 500 ppm
Sep 25 22:52:50 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376770, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:52:50.117579 21242 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:52:50.118456 21242 webserver.cc:460] Webserver started at http://127.4.219.131:46629/ using document root <none> and password file <none>
I20260925 22:52:50.119014 21242 fs_manager.cc:359] Metadata directory not provided
I20260925 22:52:50.119119 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:52:50.123046 21242 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 22:52:50.125077 21257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:50.126292 21242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:52:50.126422 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "58d5985b20224a43823526246b334235"
format_stamp: "Formatted at 2026-09-25 22:52:29 on dist-test-slave-kb1g"
server_key: "b5f05fc66589ecb648152bc5f99ee21c"
server_key_iv: "26bd213516a1f61ab0d9918694e738b0"
server_key_version: "encryptionkey@0"
I20260925 22:52:50.127005 21242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:52:50.155668 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:52:50.158628 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:52:50.158830 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:52:50.159425 21242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:52:50.160284 21242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:52:50.160356 21242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:50.160427 21242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:52:50.160478 21242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:52:50.169183 21242 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:39539
I20260925 22:52:50.169205 21370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:39539 every 8 connection(s)
I20260925 22:52:50.170133 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:52:50.172480 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21242
Sep 25 22:52:50 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376770, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:52:50.181022 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.173120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.4.219.190:34531)
0925 22:52:50.173363 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.173367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.173534 (+ 167us) server_negotiation.cc:407] Connection header received
0925 22:52:50.173643 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.173646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.173696 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.173786 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:50.174468 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.175015 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.175867 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.176100 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.176209 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.177596 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:50.177600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:50.177602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:50.177635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:50.179276 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.179719 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.179722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.179724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.179812 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.180086 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.180090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.180092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.180294 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:50.180505 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.180701 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.180847 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"spinlock_wait_cycles":18048,"thread_start_us":70,"threads_started":1}
I20260925 22:52:50.183946 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.171826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:50821 (local address 127.4.219.190:34531)
0925 22:52:50.171996 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.172001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.173098 (+ 1097us) server_negotiation.cc:407] Connection header received
0925 22:52:50.174231 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.174234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.174277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.174343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:50.175431 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.175912 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.177123 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.177337 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.177443 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.180131 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:50.180143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:50.180147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:50.180193 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:50.181932 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.182410 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.182415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.182416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.182493 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.182939 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.182944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.182945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.183103 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:50.183213 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.183670 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.183798 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"spinlock_wait_cycles":1280}
I20260925 22:52:50.184785 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.172227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:50821)
0925 22:52:50.172951 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.173002 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:50.174044 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:50.174513 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:50.174520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:50.174850 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:52:50.175298 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.175311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.176033 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.176036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.177001 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.177010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.177482 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.177486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.177650 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.178315 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:52:50.178344 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:52:50.179976 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:52:50.182037 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:50.182043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:50.182057 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:50.182308 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:50.182711 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:52:50.182713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:52:50.182715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:52:50.182841 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:52:50.183212 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:52:50.183218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:52:50.183447 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 22:52:50.183630 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 22:52:50.183952 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":138,"threads_started":1}
I20260925 22:52:50.185789 21371 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34531
I20260925 22:52:50.186026 21371 heartbeater.cc:464] Registering TS with master...
I20260925 22:52:50.186503 21371 heartbeater.cc:511] Master 127.4.219.190:34531 requested a full tablet report, sending...
I20260925 22:52:50.187455 20913 ts_manager.cc:194] Registered new tserver with Master: 58d5985b20224a43823526246b334235 (127.4.219.131:39539)
I20260925 22:52:50.188000 20913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:50821
I20260925 22:52:50.194258 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:52:50.198494 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.194878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.4.219.190:34531)
0925 22:52:50.195029 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.195033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.195206 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:52:50.195400 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.195402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.195493 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.195601 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:50.195844 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.196341 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.197271 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.197490 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.197602 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.197915 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:50.198043 (+ 128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:52:50.198331 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53406: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:52:50.198652 21103 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53406: token expired
W20260925 22:52:50.199134 20839 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.219.190:34531)
I20260925 22:52:50.207801 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.199562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.4.219.190:34531)
0925 22:52:50.199816 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.199819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.199828 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:52:50.199854 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.199855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.199886 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.199986 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:52:50.200762 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.201564 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.202890 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.203085 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.203159 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.204950 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:52:50.204953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:52:50.204956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:52:50.204977 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:52:50.206392 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.206767 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.206770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.206771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.206838 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:52:50.207054 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:52:50.207056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:52:50.207057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:52:50.207210 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:52:50.207306 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.207491 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.207572 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260925 22:52:50.210181 20913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:52:50.213395 20913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:52:50.221283 21035 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7ec18092084812b36637f448ced2ca (DEFAULT_TABLE table=test-table [id=787e4ec2659f44d486a442ebcd9df9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:50.221948 21035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7ec18092084812b36637f448ced2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:50.225111 20993 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap starting.
I20260925 22:52:50.225772 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.220067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.0.0.1:39208)
0925 22:52:50.220347 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.220359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:50.220466 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:50.221286 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:50.221290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:50.221312 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:50.221446 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.221455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.222970 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.222973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.224083 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.224093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.225244 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.225247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.225377 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.225423 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:52:50.225494 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:52:50.225606 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":48,"thread_start_us":124,"threads_started":1}
I20260925 22:52:50.225805 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.220068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.4.219.131:39539)
0925 22:52:50.220840 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.220851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.220871 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:52:50.220934 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.220938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.221108 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.221281 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:50.221626 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.222777 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.223895 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.225126 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.225366 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.225513 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.225568 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.225656 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":143,"threads_started":1}
I20260925 22:52:50.225873 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.219786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:39539 (local address 127.0.0.1:57812)
0925 22:52:50.219939 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.219972 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:50.220050 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:50.221257 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:50.221260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:50.221277 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:50.221425 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.221434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.222894 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.222901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.223762 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.223771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.225262 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.225266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.225424 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.225441 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:52:50.225497 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 22:52:50.225731 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":216,"spinlock_wait_cycles":896}
I20260925 22:52:50.226020 20993 tablet_bootstrap.cc:654] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:50.226882 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.220421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.4.219.130:38155)
0925 22:52:50.220804 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.220821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.220846 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:52:50.220925 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.220931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.221129 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.221584 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:50.221779 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.222819 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.224237 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.225116 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.226366 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.226559 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.226618 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.226720 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":204,"threads_started":1}
I20260925 22:52:50.227551 20993 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: No bootstrap required, opened a new log
I20260925 22:52:50.227640 20993 ts_tablet_manager.cc:1397] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:52:50.228091 20993 raft_consensus.cc:348] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.228212 20993 raft_consensus.cc:374] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:50.228250 20993 raft_consensus.cc:729] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6ff17d29af49afb1a13921064c6d38, State: Initialized, Role: FOLLOWER
I20260925 22:52:50.228374 20993 consensus_queue.cc:260] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.228684 20993 ts_tablet_manager.cc:1428] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:52:50.229017 21305 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7ec18092084812b36637f448ced2ca (DEFAULT_TABLE table=test-table [id=787e4ec2659f44d486a442ebcd9df9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:50.230113 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7ec18092084812b36637f448ced2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:50.230006 21173 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7ec18092084812b36637f448ced2ca (DEFAULT_TABLE table=test-table [id=787e4ec2659f44d486a442ebcd9df9e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:52:50.231015 21173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7ec18092084812b36637f448ced2ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:52:50.236899 21384 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Bootstrap starting.
I20260925 22:52:50.237330 21385 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Bootstrap starting.
I20260925 22:52:50.239738 21385 tablet_bootstrap.cc:654] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:50.239766 21384 tablet_bootstrap.cc:654] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Neither blocks nor log segments found. Creating new log.
I20260925 22:52:50.240481 21385 log.cc:824] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:50.240492 21384 log.cc:824] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Log is configured to *not* fsync() on all Append() calls
I20260925 22:52:50.242251 21385 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: No bootstrap required, opened a new log
I20260925 22:52:50.242264 21384 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: No bootstrap required, opened a new log
I20260925 22:52:50.242441 21384 ts_tablet_manager.cc:1397] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 22:52:50.242445 21385 ts_tablet_manager.cc:1397] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:52:50.245505 21384 raft_consensus.cc:348] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.245505 21385 raft_consensus.cc:348] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.245744 21385 raft_consensus.cc:374] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:50.245750 21384 raft_consensus.cc:374] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:52:50.245834 21385 raft_consensus.cc:729] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fce4008f5e47dfb0a109d64063b97c, State: Initialized, Role: FOLLOWER
I20260925 22:52:50.245849 21384 raft_consensus.cc:729] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d5985b20224a43823526246b334235, State: Initialized, Role: FOLLOWER
I20260925 22:52:50.246268 21384 consensus_queue.cc:260] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.246268 21385 consensus_queue.cc:260] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.246887 21371 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
I20260925 22:52:50.246887 21239 heartbeater.cc:503] Master 127.4.219.190:34531 was elected leader, sending a full tablet report...
I20260925 22:52:50.247229 21385 ts_tablet_manager.cc:1428] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:52:50.247231 21384 ts_tablet_manager.cc:1428] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:52:50.254724 21388 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:52:50.254892 21388 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.255910 21388 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:50.260454 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.256156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:52965 (local address 127.4.219.130:38155)
0925 22:52:50.256299 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.256303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.256318 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:52:50.256446 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.256450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.256506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.256626 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:50.256916 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.257698 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.259249 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.259852 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.259975 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.260037 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.260270 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.260335 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:52:50.260456 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.256320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:52655 (local address 127.4.219.129:38325)
0925 22:52:50.256446 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.256451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.256686 (+ 235us) server_negotiation.cc:407] Connection header received
0925 22:52:50.256775 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.256777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.256825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.256932 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:52:50.257162 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.257931 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.259277 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.259948 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.260051 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.260182 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.260268 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.260335 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:52:50.260555 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.256072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.4.219.131:52965)
0925 22:52:50.256241 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.256257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:50.256341 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:50.256651 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:50.256656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:50.256674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:50.256806 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.256814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.257832 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.257836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.259111 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.259122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.259980 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.259983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.260128 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.260166 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 22:52:50.260248 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:52:50.260401 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":45}
I20260925 22:52:50.260572 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.256141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.4.219.131:52655)
0925 22:52:50.256584 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.256598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:52:50.256690 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:52:50.256953 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:52:50.256955 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:52:50.256969 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:52:50.257057 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.257061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.258034 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.258038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.259171 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:52:50.259180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.260055 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.260058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:52:50.260148 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.260173 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:52:50.260248 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:52:50.260377 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":42,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260925 22:52:50.261157 21193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7ec18092084812b36637f448ced2ca" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c" is_pre_election: true
I20260925 22:52:50.261327 21055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7ec18092084812b36637f448ced2ca" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6ff17d29af49afb1a13921064c6d38" is_pre_election: true
I20260925 22:52:50.261421 21193 raft_consensus.cc:2454] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d5985b20224a43823526246b334235 in term 0.
I20260925 22:52:50.261592 21055 raft_consensus.cc:2454] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d5985b20224a43823526246b334235 in term 0.
I20260925 22:52:50.261879 21260 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c; no voters:
I20260925 22:52:50.262130 21388 raft_consensus.cc:2781] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:52:50.262219 21388 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:52:50.262262 21388 raft_consensus.cc:3037] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:50.263185 21388 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
W20260925 22:52:50.263473 21102 tablet.cc:2369] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:52:50.263511 21388 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 election: Requested vote from peers f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155), 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325)
I20260925 22:52:50.263777 21193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7ec18092084812b36637f448ced2ca" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3fce4008f5e47dfb0a109d64063b97c"
I20260925 22:52:50.263806 21055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7ec18092084812b36637f448ced2ca" candidate_uuid: "58d5985b20224a43823526246b334235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6ff17d29af49afb1a13921064c6d38"
I20260925 22:52:50.263878 21193 raft_consensus.cc:3037] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:50.263913 21055 raft_consensus.cc:3037] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:52:50.264941 21193 raft_consensus.cc:2454] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d5985b20224a43823526246b334235 in term 1.
I20260925 22:52:50.264957 21055 raft_consensus.cc:2454] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d5985b20224a43823526246b334235 in term 1.
I20260925 22:52:50.265247 21258 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58d5985b20224a43823526246b334235, 9d6ff17d29af49afb1a13921064c6d38; no voters:
I20260925 22:52:50.265421 21388 raft_consensus.cc:2781] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:52:50.265602 21388 raft_consensus.cc:686] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 1 LEADER]: Becoming Leader. State: Replica: 58d5985b20224a43823526246b334235, State: Running, Role: LEADER
I20260925 22:52:50.265926 21388 consensus_queue.cc:237] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:52:50.269105 20913 catalog_manager.cc:5495] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d5985b20224a43823526246b334235 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "58d5985b20224a43823526246b334235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } health_report { overall_health: UNKNOWN } } }
I20260925 22:52:50.289093 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.285772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.4.219.131:39539)
0925 22:52:50.285903 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:52:50.285907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:52:50.285920 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:52:50.286011 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:52:50.286013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:52:50.286066 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:52:50.286154 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:52:50.286368 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.286874 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.287635 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:52:50.287846 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:52:50.287919 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:52:50.288172 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:52:50.288689 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:52:50.288785 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:52:50.288841 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:52:50.288889 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:52:50.294998 21055 raft_consensus.cc:1261] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Refusing update from remote peer 58d5985b20224a43823526246b334235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:50.295001 21193 raft_consensus.cc:1261] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Refusing update from remote peer 58d5985b20224a43823526246b334235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:52:50.295769 21391 consensus_queue.cc:1035] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:50.295994 21388 consensus_queue.cc:1035] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:52:50.303980 21396 mvcc.cc:205] Tried to move back new op lower bound from 7333383251121389568 to 7333383251011743744. Current Snapshot: MvccSnapshot[applied={T|T < 7333383251121389568}]
I20260925 22:52:50.310187 20908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:50.310366 20908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:52:50.312680 20908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Sending DeleteTablet for 3 replicas of tablet 2a7ec18092084812b36637f448ced2ca
I20260925 22:52:50.313344 21305 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7ec18092084812b36637f448ced2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:50 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:57812
I20260925 22:52:50.313442 21173 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7ec18092084812b36637f448ced2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:50 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:39208
I20260925 22:52:50.313509 21035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7ec18092084812b36637f448ced2ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:52:50 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:49356
I20260925 22:52:50.313848 21404 tablet_replica.cc:330] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: stopping tablet replica
I20260925 22:52:50.314091 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20951
I20260925 22:52:50.314642 21404 raft_consensus.cc:2229] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:52:50.315083 21404 raft_consensus.cc:2258] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:50.316330 21406 tablet_replica.cc:330] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: stopping tablet replica
I20260925 22:52:50.316672 21406 raft_consensus.cc:2229] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:52:50.316978 21406 raft_consensus.cc:2258] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:52:50.317149 21404 ts_tablet_manager.cc:1905] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:50.318557 21406 ts_tablet_manager.cc:1905] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:52:50.322113 21406 ts_tablet_manager.cc:1918] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:50.322247 21406 log.cc:1196] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/2a7ec18092084812b36637f448ced2ca
I20260925 22:52:50.322652 21406 ts_tablet_manager.cc:1939] T 2a7ec18092084812b36637f448ced2ca P f3fce4008f5e47dfb0a109d64063b97c: Deleting consensus metadata
W20260925 22:52:50.323129 20897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:38325 (error 108)
I20260925 22:52:50.324066 20899 catalog_manager.cc:4841] TS f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): tablet 2a7ec18092084812b36637f448ced2ca (table test-table [id=787e4ec2659f44d486a442ebcd9df9e1]) successfully deleted
I20260925 22:52:50.324725 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21105
I20260925 22:52:50.325093 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:52:50.324718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:38325 (local address 127.0.0.1:49368)
0925 22:52:50.324916 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:52:50.324993 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 22:52:50.325277 21404 ts_tablet_manager.cc:1918] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:52:50.325407 21404 log.cc:1196] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/2a7ec18092084812b36637f448ced2ca
I20260925 22:52:50.325793 21404 ts_tablet_manager.cc:1939] T 2a7ec18092084812b36637f448ced2ca P 58d5985b20224a43823526246b334235: Deleting consensus metadata
W20260925 22:52:50.326161 20897 catalog_manager.cc:4584] TS 9d6ff17d29af49afb1a13921064c6d38 (127.4.219.129:38325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a7ec18092084812b36637f448ced2ca: Network error: Client connection negotiation failed: client connection to 127.4.219.129:38325: connect: Connection refused (error 111)
I20260925 22:52:50.327255 20900 catalog_manager.cc:4841] TS 58d5985b20224a43823526246b334235 (127.4.219.131:39539): tablet 2a7ec18092084812b36637f448ced2ca (table test-table [id=787e4ec2659f44d486a442ebcd9df9e1]) successfully deleted
I20260925 22:52:50.333498 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21242
I20260925 22:52:50.339174 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 20880
I20260925 22:53:10.346764 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--webserver_interface=127.4.219.190
--webserver_port=43735
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:34531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:10.456983 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:10.457262 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:10.457305 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:10.461376 21408 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:53:10.461439 21408 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:53:10.461455 21408 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:53:10.461469 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:10.461488 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:10.461530 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:10.461563 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:10.461591 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:10.466945 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:34531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=43735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:10.468065 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:10.469169 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:10.477629 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:10.477645 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.477690 21408 server_base.cc:1034] running on GCE node
W20260925 22:53:10.477644 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.478497 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:10.479650 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:10.480834 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790376790480819 us; error 30 us; skew 500 ppm
Sep 25 22:53:10 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376790, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:10.487197 21408 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:10.488142 21408 webserver.cc:460] Webserver started at http://127.4.219.190:43735/ using document root <none> and password file <none>
I20260925 22:53:10.488792 21408 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:10.488883 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:10.492918 21408 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:10.495108 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:10.496387 21408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:10.496575 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "45aa2142d58442c99c0016f3dc78d435"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "67ef7689ff6c14d8f377456cb76cd3b8"
server_key_iv: "70b4e5fe3c6e6904dd93286949dd9fa4"
server_key_version: "encryptionkey@0"
I20260925 22:53:10.497150 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:10.508816 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:10.512040 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:10.512246 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:10.519436 21408 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:34531
I20260925 22:53:10.519791 21475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:34531 every 8 connection(s)
I20260925 22:53:10.520524 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:10.522935 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21408
I20260925 22:53:10.523370 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:38325
--local_ip_for_outbound_sockets=127.4.219.129
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=44575
--webserver_interface=127.4.219.129
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:53:10.525111 21476 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:53:10.527539 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap starting.
I20260925 22:53:10.551873 21476 log.cc:824] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:10.565955 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:53:10.566412 21476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Bootstrap complete.
I20260925 22:53:10.570994 21476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:53:10.571420 21476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Initialized, Role: FOLLOWER
I20260925 22:53:10.571903 21476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:53:10.572046 21476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:10.572134 21476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:10.572252 21476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:53:10.574195 21476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:53:10.574532 21476 leader_election.cc:302] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45aa2142d58442c99c0016f3dc78d435; no voters:
I20260925 22:53:10.574844 21476 leader_election.cc:288] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:53:10.574968 21480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:53:10.575208 21480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [term 3 LEADER]: Becoming Leader. State: Replica: 45aa2142d58442c99c0016f3dc78d435, State: Running, Role: LEADER
I20260925 22:53:10.575582 21480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } }
I20260925 22:53:10.576043 21476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:10.577147 21481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:53:10.577150 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45aa2142d58442c99c0016f3dc78d435. Latest consensus state: current_term: 3 leader_uuid: "45aa2142d58442c99c0016f3dc78d435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45aa2142d58442c99c0016f3dc78d435" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34531 } } }
I20260925 22:53:10.577286 21481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:10.577292 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:10.577641 21489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:10.582767 21489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:10.583689 21489 catalog_manager.cc:1259] Loaded cluster ID: 5c5caadd8a474ccfaa4f41165b0ad880
I20260925 22:53:10.583760 21489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:10.586238 21489 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:10.586930 21489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Loaded TSK: 0
I20260925 22:53:10.587011 21489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45aa2142d58442c99c0016f3dc78d435: Loaded TSK: 1
F20260925 22:53:10.587702 21489 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5c6594e9ad google::LogMessage::Fail() at ??:0
@ 0x7f5c6595188a google::LogMessage::SendToLog() at ??:0
@ 0x7f5c6594e4de google::LogMessage::Flush() at ??:0
@ 0x7f5c6594fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5c6861218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5c6860af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5c702b37bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5c7028d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5c7028e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f5c702cfdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f5c702ee82b std::function<>::operator()() at ??:0
@ 0x7f5c7028db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5c7028ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5c7028b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5c702cf640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5c67ba03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5c67ba0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5c67ba23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5c67b91512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5c691d56db start_thread at ??:0
@ 0x7f5c6444c61f clone at ??:0
W20260925 22:53:10.678303 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:10.678517 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:10.678560 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:10.682958 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:10.683009 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:10.683068 21478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:10.688274 21478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:38325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=44575
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:10.689405 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:10.690441 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:10.700001 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:10.700001 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.700263 21478 server_base.cc:1034] running on GCE node
W20260925 22:53:10.700053 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.700790 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:10.701464 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:10.702641 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790376790702619 us; error 49 us; skew 500 ppm
Sep 25 22:53:10 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376790, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:10.705930 21478 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:53:10.706833 21478 webserver.cc:460] Webserver started at http://127.4.219.129:44575/ using document root <none> and password file <none>
I20260925 22:53:10.707410 21478 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:10.707499 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:10.711647 21478 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:53:10.714087 21533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:10.715466 21478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:53:10.715595 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "9d6ff17d29af49afb1a13921064c6d38"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "106c88c40e3659eb74ddd053c41f6ea8"
server_key_iv: "c8051b5cead7595a614cce82fd5e87f5"
server_key_version: "encryptionkey@0"
I20260925 22:53:10.716161 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:10.739292 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:10.742751 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:10.742950 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:10.743687 21478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:10.744833 21540 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:53:10.748865 21478 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:53:10.748924 21478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 22:53:10.748986 21478 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:53:10.750597 21478 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:53:10.750651 21478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:53:10.750916 21540 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap starting.
I20260925 22:53:10.761130 21478 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:38325
I20260925 22:53:10.761314 21647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:38325 every 8 connection(s)
I20260925 22:53:10.763649 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:10.771027 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21478
I20260925 22:53:10.771410 21540 log.cc:824] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:10.771525 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:38155
--local_ip_for_outbound_sockets=127.4.219.130
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=37833
--webserver_interface=127.4.219.130
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:53:10.778909 21540 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:53:10.779395 21540 tablet_bootstrap.cc:492] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Bootstrap complete.
I20260925 22:53:10.780504 21540 ts_tablet_manager.cc:1397] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.000s
I20260925 22:53:10.782821 21540 raft_consensus.cc:348] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:10.783234 21540 raft_consensus.cc:729] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6ff17d29af49afb1a13921064c6d38, State: Initialized, Role: FOLLOWER
I20260925 22:53:10.783706 21540 consensus_queue.cc:260] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:10.784659 21540 ts_tablet_manager.cc:1428] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260925 22:53:10.829769 21650 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:53:10.766443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:56351)
0925 22:53:10.767070 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.767127 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:10.768286 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:10.828822 (+ 60536us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":142,"threads_started":1}
W20260925 22:53:10.830394 21648 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:34531 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:34531: Client connection negotiation failed: client connection to 127.4.219.190:34531: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:53:10.831238 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:10.830893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:40389)
0925 22:53:10.831059 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.831150 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:53:10.831966 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:10.831719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:56425)
0925 22:53:10.831847 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.831910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:53:10.832171 21648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:53:10.899922 21652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:10.900175 21652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:10.900264 21652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:10.904229 21652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:10.904284 21652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:10.904378 21652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:10.909528 21652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:38155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=37833
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:10.910555 21652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:10.911597 21652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:10.921078 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:10.921093 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.921317 21652 server_base.cc:1034] running on GCE node
W20260925 22:53:10.921078 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:10.921832 21652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:10.922474 21652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:10.923676 21652 hybrid_clock.cc:648] HybridClock initialized: now 1790376790923643 us; error 29 us; skew 500 ppm
Sep 25 22:53:10 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376790, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:10.926569 21652 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:53:10.927526 21652 webserver.cc:460] Webserver started at http://127.4.219.130:37833/ using document root <none> and password file <none>
I20260925 22:53:10.928155 21652 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:10.928252 21652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:10.932260 21652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:53:10.934338 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:10.935549 21652 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 22:53:10.935683 21652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "f3fce4008f5e47dfb0a109d64063b97c"
format_stamp: "Formatted at 2026-09-25 22:52:28 on dist-test-slave-kb1g"
server_key: "ad836e7769dd118eb5279a857b4e7c02"
server_key_iv: "0e03a7a480d51bcd9fa7e4490b60cb51"
server_key_version: "encryptionkey@0"
I20260925 22:53:10.936231 21652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:10.949658 21652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:10.952495 21652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:10.952729 21652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:10.953392 21652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:10.954139 21652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:10.954183 21652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:10.954229 21652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:10.954250 21652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:10.962815 21652 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:38155
I20260925 22:53:10.962836 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:38155 every 8 connection(s)
I20260925 22:53:10.963855 21652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:10.967295 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:10.965888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:54915)
0925 22:53:10.966501 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.966601 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":108,"threads_started":1}
W20260925 22:53:10.967725 21782 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:34531 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:34531: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
I20260925 22:53:10.968338 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:10.968122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:46821)
0925 22:53:10.968232 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.968282 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 22:53:10.968998 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:10.968766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:33413)
0925 22:53:10.968893 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:10.968947 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:53:10.969189 21782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:53:10.971776 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21652
I20260925 22:53:10.972232 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:39539
--local_ip_for_outbound_sockets=127.4.219.131
--tserver_master_addrs=127.4.219.190:34531
--webserver_port=46629
--webserver_interface=127.4.219.131
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:11.082679 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:11.082944 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:11.082988 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:11.087061 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:11.087127 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:11.087190 21785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:11.092408 21785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:39539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=46629
--tserver_master_addrs=127.4.219.190:34531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:11.093513 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:11.094527 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:11.104069 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:11.104084 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:11.104084 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:11.104353 21785 server_base.cc:1034] running on GCE node
I20260925 22:53:11.104907 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:11.105551 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:11.106714 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790376791106682 us; error 44 us; skew 500 ppm
Sep 25 22:53:11 dist-test-slave-kb1g krb5kdc[20329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376791, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:11.109609 21785 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:53:11.110481 21785 webserver.cc:460] Webserver started at http://127.4.219.131:46629/ using document root <none> and password file <none>
I20260925 22:53:11.110999 21785 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:11.111058 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:11.114836 21785 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:11.116871 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:11.118291 21785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:53:11.118389 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "58d5985b20224a43823526246b334235"
format_stamp: "Formatted at 2026-09-25 22:52:29 on dist-test-slave-kb1g"
server_key: "b5f05fc66589ecb648152bc5f99ee21c"
server_key_iv: "26bd213516a1f61ab0d9918694e738b0"
server_key_version: "encryptionkey@0"
I20260925 22:53:11.118898 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:11.155827 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:11.159235 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:11.159390 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:11.159951 21785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:11.160797 21785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:11.160845 21785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:11.160894 21785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:11.160920 21785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:11.169700 21785 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:39539
I20260925 22:53:11.169780 21913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:39539 every 8 connection(s)
I20260925 22:53:11.170725 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:11.174444 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.172941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:43001)
0925 22:53:11.173599 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.173722 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":111,"threads_started":1}
W20260925 22:53:11.174899 21914 heartbeater.cc:644] Failed to heartbeat to 127.4.219.190:34531 (0 consecutive failures): Network error: Failed to ping master at 127.4.219.190:34531: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
I20260925 22:53:11.175451 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.175229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:33937)
0925 22:53:11.175351 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.175399 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 22:53:11.176097 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.175922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:45571)
0925 22:53:11.176013 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.176055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 22:53:11.176299 21914 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:53:11.179000 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21785
I20260925 22:53:11.833372 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.832928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:41567)
0925 22:53:11.833188 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.833251 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260925 22:53:11.932794 21919 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:11.933037 21919 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:11.934180 21919 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d5985b20224a43823526246b334235 (127.4.219.131:39539), f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
Sep 25 22:53:11 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376790, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.131@KRBTEST.COM
Sep 25 22:53:11 dist-test-slave-kb1g krb5kdc[20329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376790, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.130@KRBTEST.COM
I20260925 22:53:11.947865 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.934446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:39539 (local address 127.4.219.129:49499)
0925 22:53:11.934970 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.934983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:11.935070 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:11.936641 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:11.936644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:11.936958 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:11.937437 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:11.937451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.938683 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.938687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:11.939803 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:11.939812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.940853 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.940856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:11.940916 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:11.941470 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:11.941484 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:11.943055 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:11.945549 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:11.945561 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:11.945574 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:11.945817 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:11.946149 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:11.946154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:11.946155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:11.946302 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:11.946900 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:11.946906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:11.947133 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 22:53:11.947313 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:53:11.947669 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":315,"spinlock_wait_cycles":4352,"thread_start_us":71,"threads_started":1}
I20260925 22:53:11.947875 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.934671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:49499 (local address 127.4.219.131:39539)
0925 22:53:11.934990 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:11.935002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:11.935041 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:53:11.935793 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:11.935798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:11.936099 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:11.936358 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:11.937603 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.938554 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.939922 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.940524 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.940709 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:11.943287 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:11.943292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:11.943303 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:11.943330 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:11.945439 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:11.945929 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:11.945935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:11.945941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:11.946059 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:11.946392 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:11.946397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:11.946399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:11.946788 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:11.946924 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:11.947418 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 22:53:11.947673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260925 22:53:11.947880 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.934569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:39169 (local address 127.4.219.130:38155)
0925 22:53:11.935027 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:11.935042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:11.935079 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:53:11.935753 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:11.935759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:11.936099 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:11.936377 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:11.937603 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.938542 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.939913 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.940507 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.940700 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:11.943611 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:11.943618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:11.943633 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:11.943667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:11.945718 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:11.946136 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:11.946141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:11.946147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:11.946258 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:11.946530 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:11.946533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:11.946535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:11.946881 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:11.947013 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:11.947418 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 22:53:11.947689 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":115,"threads_started":1}
I20260925 22:53:11.947897 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.934446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38155 (local address 127.4.219.129:39169)
0925 22:53:11.934612 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.934628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:11.934752 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:11.936434 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:11.936440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:11.936969 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:11.937438 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:11.937451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.938678 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.938681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:11.939801 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:11.939810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:11.940645 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:11.940648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:11.940806 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:11.941532 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:11.941543 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:11.943398 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:11.945828 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:11.945833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:11.945835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:11.946025 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:11.946352 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:11.946356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:11.946358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:11.946439 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:11.946986 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:11.946989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:11.947134 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 22:53:11.947313 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 22:53:11.947746 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":288,"spinlock_wait_cycles":57472}
W20260925 22:53:11.950778 21537 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d5985b20224a43823526246b334235 (127.4.219.131:39539): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
W20260925 22:53:11.950958 21536 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
I20260925 22:53:11.951051 21536 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38; no voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:53:11.951315 21919 raft_consensus.cc:2726] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:53:11.970054 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:11.969693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:60489)
0925 22:53:11.969917 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:11.969967 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":64,"threads_started":1}
I20260925 22:53:12.177309 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:12.176877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:44367)
0925 22:53:12.177147 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:12.177212 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260925 22:53:12.834561 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:12.834141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:57631)
0925 22:53:12.834397 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:12.834462 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260925 22:53:12.971185 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:12.970782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:59723)
0925 22:53:12.971025 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:12.971078 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260925 22:53:13.178566 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:13.178106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:50667)
0925 22:53:13.178404 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:13.178459 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260925 22:53:13.656816 21928 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:13.656951 21928 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:13.657457 21928 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d5985b20224a43823526246b334235 (127.4.219.131:39539), f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
W20260925 22:53:13.658356 21537 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d5985b20224a43823526246b334235 (127.4.219.131:39539): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
W20260925 22:53:13.658437 21536 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
I20260925 22:53:13.658492 21537 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38; no voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:53:13.658651 21928 raft_consensus.cc:2726] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:53:13.835695 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:13.835247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:48045)
0925 22:53:13.835545 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:13.835600 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
I20260925 22:53:13.972393 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:13.971975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:37791)
0925 22:53:13.972246 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:13.972302 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260925 22:53:14.179723 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:14.179311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:35641)
0925 22:53:14.179574 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:14.179632 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260925 22:53:14.836897 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:14.836477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:34291)
0925 22:53:14.836735 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:14.836792 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":90,"threads_started":1}
I20260925 22:53:14.973598 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:14.973159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:46117)
0925 22:53:14.973418 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:14.973481 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
I20260925 22:53:15.180996 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:15.180521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:48649)
0925 22:53:15.180834 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:15.180897 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":73,"threads_started":1}
I20260925 22:53:15.838363 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:15.837862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:36755)
0925 22:53:15.838200 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:15.838267 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":75,"threads_started":1}
I20260925 22:53:15.974928 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:15.974518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:39439)
0925 22:53:15.974783 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:15.974840 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260925 22:53:16.182258 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:16.181835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:42185)
0925 22:53:16.182101 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:16.182163 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260925 22:53:16.744944 21938 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:16.745097 21938 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:16.745539 21938 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d5985b20224a43823526246b334235 (127.4.219.131:39539), f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
W20260925 22:53:16.746362 21537 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d5985b20224a43823526246b334235 (127.4.219.131:39539): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
W20260925 22:53:16.746456 21536 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
I20260925 22:53:16.746600 21536 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38; no voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:53:16.746776 21938 raft_consensus.cc:2726] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:53:16.839715 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:16.839257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:49157)
0925 22:53:16.839554 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:16.839617 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260925 22:53:16.976117 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:16.975694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:45381)
0925 22:53:16.975962 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:16.976025 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260925 22:53:17.183518 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:17.183036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:59467)
0925 22:53:17.183331 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:17.183402 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
I20260925 22:53:17.840991 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:17.840448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:44017)
0925 22:53:17.840835 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:17.840893 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":152,"threads_started":1}
I20260925 22:53:17.977321 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:17.976899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:41057)
0925 22:53:17.977182 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:17.977234 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
I20260925 22:53:18.184790 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:18.184269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:43403)
0925 22:53:18.184631 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:18.184700 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":82,"threads_started":1}
I20260925 22:53:18.842060 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:18.841660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:46591)
0925 22:53:18.841911 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:18.841964 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260925 22:53:18.978425 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:18.978029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:33303)
0925 22:53:18.978277 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:18.978332 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260925 22:53:19.186012 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:19.185589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:42911)
0925 22:53:19.185855 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:19.185916 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260925 22:53:19.252017 21948 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:19.252180 21948 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:19.253214 21948 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d5985b20224a43823526246b334235 (127.4.219.131:39539), f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
W20260925 22:53:19.253331 21536 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
W20260925 22:53:19.253476 21537 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d5985b20224a43823526246b334235 (127.4.219.131:39539): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
I20260925 22:53:19.253554 21537 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38; no voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:53:19.253716 21948 raft_consensus.cc:2726] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:53:19.843206 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:19.842792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:56485)
0925 22:53:19.843048 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:19.843107 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260925 22:53:19.979671 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:19.979189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:43439)
0925 22:53:19.979481 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:19.979539 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260925 22:53:20.187145 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:20.186706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:53355)
0925 22:53:20.186965 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:20.187052 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260925 22:53:20.844520 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:20.844141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:51975)
0925 22:53:20.844368 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:20.844425 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260925 22:53:20.980834 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:20.980414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:51687)
0925 22:53:20.980689 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:20.980742 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":66,"threads_started":1}
I20260925 22:53:21.188408 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:21.187981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:48789)
0925 22:53:21.188248 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:21.188309 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260925 22:53:21.845763 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:21.845316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:36127)
0925 22:53:21.845600 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:21.845664 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260925 22:53:21.979502 21956 raft_consensus.cc:482] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:21.979666 21956 raft_consensus.cc:504] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58d5985b20224a43823526246b334235" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 39539 } } peers { permanent_uuid: "f3fce4008f5e47dfb0a109d64063b97c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38155 } } peers { permanent_uuid: "9d6ff17d29af49afb1a13921064c6d38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 38325 } }
I20260925 22:53:21.980067 21956 leader_election.cc:288] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58d5985b20224a43823526246b334235 (127.4.219.131:39539), f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155)
W20260925 22:53:21.980885 21537 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58d5985b20224a43823526246b334235 (127.4.219.131:39539): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
W20260925 22:53:21.981019 21536 leader_election.cc:341] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f3fce4008f5e47dfb0a109d64063b97c (127.4.219.130:38155): Not found: Tablet not found: 2a7ec18092084812b36637f448ced2ca
I20260925 22:53:21.981117 21536 leader_election.cc:302] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d6ff17d29af49afb1a13921064c6d38; no voters: 58d5985b20224a43823526246b334235, f3fce4008f5e47dfb0a109d64063b97c
I20260925 22:53:21.981263 21956 raft_consensus.cc:2726] T 2a7ec18092084812b36637f448ced2ca P 9d6ff17d29af49afb1a13921064c6d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:53:21.981895 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:21.981537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:32779)
0925 22:53:21.981752 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:21.981813 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260925 22:53:22.189630 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:22.189202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:52135)
0925 22:53:22.189458 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:22.189517 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":89,"threads_started":1}
I20260925 22:53:22.846900 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:22.846483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:58323)
0925 22:53:22.846747 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:22.846803 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260925 22:53:22.983065 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:22.982655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:45259)
0925 22:53:22.982906 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:22.982966 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260925 22:53:23.190822 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:23.190411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:56027)
0925 22:53:23.190663 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:23.190726 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260925 22:53:23.848102 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:23.847656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:32831)
0925 22:53:23.847938 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:23.847996 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260925 22:53:23.984298 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:23.983876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:39637)
0925 22:53:23.984143 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:23.984201 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":73,"threads_started":1}
I20260925 22:53:24.192067 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:24.191612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:52993)
0925 22:53:24.191899 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:24.191965 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":73,"threads_started":1}
I20260925 22:53:24.849395 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:24.848978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:38783)
0925 22:53:24.849234 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:24.849295 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260925 22:53:24.985445 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:24.985017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:36403)
0925 22:53:24.985294 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:24.985349 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
I20260925 22:53:25.193418 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:25.192975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.131:46233)
0925 22:53:25.193267 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:25.193323 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
I20260925 22:53:25.850757 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:25.850354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.129:44001)
0925 22:53:25.850609 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:25.850663 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260925 22:53:25.986706 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:25.986290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34531 (local address 127.4.219.130:49089)
0925 22:53:25.986554 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:25.986608 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.190:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260925 22:53:26.180645 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21478
I20260925 22:53:26.187084 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21652
I20260925 22:53:26.192688 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21785
2026-09-25T22:53:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59115 ms)
[----------] 30 tests from SecurityITest (239265 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:26 dist-test-slave-kb1g krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:26 dist-test-slave-kb1g krb5kdc[21972](info): set up 2 sockets
Sep 25 22:53:26 dist-test-slave-kb1g krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260925 22:53:27.232424 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:27 dist-test-slave-kb1g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:27.355638 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:46239
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:46239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:27.476938 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:27.477277 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:27.477377 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:27.482168 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:27.482270 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:27.482312 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:27.482378 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:27.482465 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:27.488369 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:46239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:46239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:27.489610 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:27.490739 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:53:27.500224 21985 server_base.cc:1034] running on GCE node
W20260925 22:53:27.500168 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:27.500152 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:27.500168 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:27.501108 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:27.502274 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:27.503487 21985 hybrid_clock.cc:648] HybridClock initialized: now 1790376807503470 us; error 30 us; skew 500 ppm
Sep 25 22:53:27 dist-test-slave-kb1g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376807, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:27.506448 21985 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:27.507349 21985 webserver.cc:460] Webserver started at http://127.4.219.190:35045/ using document root <none> and password file <none>
I20260925 22:53:27.507912 21985 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:27.508004 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:27.508236 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:27.510350 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "2ebaef1e8f354a75bd39affa0b760095"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "ba55c77930457c55a10782b71dd77611"
server_key_iv: "d515b84daca9ab6a404aa3bc391fbc7d"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.510944 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "2ebaef1e8f354a75bd39affa0b760095"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "ba55c77930457c55a10782b71dd77611"
server_key_iv: "d515b84daca9ab6a404aa3bc391fbc7d"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.514905 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:53:27.517417 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:27.518709 21985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:27.518847 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "2ebaef1e8f354a75bd39affa0b760095"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "ba55c77930457c55a10782b71dd77611"
server_key_iv: "d515b84daca9ab6a404aa3bc391fbc7d"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.518976 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:27.534417 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:27.537480 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:27.537685 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:27.544597 21985 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:46239
I20260925 22:53:27.544648 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:46239 every 8 connection(s)
I20260925 22:53:27.545555 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:27.548336 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:27.552184 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 21985
I20260925 22:53:27.552326 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:27.552664 4974 external_mini_cluster.cc:1428] Setting key 907fed531a6f567f8b2da89d37fd5c3b
I20260925 22:53:27.554230 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: Bootstrap starting.
I20260925 22:53:27.556731 22053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:27.557503 22053 log.cc:824] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:27.559341 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: No bootstrap required, opened a new log
Sep 25 22:53:27 dist-test-slave-kb1g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:27.563827 22053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } }
I20260925 22:53:27.564090 22053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:27.564152 22053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebaef1e8f354a75bd39affa0b760095, State: Initialized, Role: FOLLOWER
I20260925 22:53:27.564713 22053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } }
I20260925 22:53:27.564862 22053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:27.564950 22053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:27.565084 22053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:27.565972 22053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } }
I20260925 22:53:27.566385 22053 leader_election.cc:302] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ebaef1e8f354a75bd39affa0b760095; no voters:
I20260925 22:53:27.566738 22053 leader_election.cc:288] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:27.566821 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:27.566991 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebaef1e8f354a75bd39affa0b760095, State: Running, Role: LEADER
I20260925 22:53:27.567293 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:27.554096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.4.219.190:46239)
0925 22:53:27.554500 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:27.554518 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:27.554565 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:53:27.555179 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:27.555195 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:27.555515 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:27.555796 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:27.556716 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.557831 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.558695 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.559182 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.559352 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:27.562220 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:27.562225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:27.562240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:27.562266 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:27.563968 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:27.564459 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:27.564470 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:27.564475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:27.564663 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:27.564935 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:27.564941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:27.564943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:27.565441 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:27.565669 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:27.565932 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 22:53:27.566254 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":140,"threads_started":1}
I20260925 22:53:27.567358 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } }
I20260925 22:53:27.568001 22053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:27.570243 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebaef1e8f354a75bd39affa0b760095. Latest consensus state: current_term: 1 leader_uuid: "2ebaef1e8f354a75bd39affa0b760095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } } }
I20260925 22:53:27.570412 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ebaef1e8f354a75bd39affa0b760095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ebaef1e8f354a75bd39affa0b760095" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 46239 } } }
I20260925 22:53:27.570675 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:27.570827 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095 [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:27.573446 22072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:27.573524 22072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:27.573583 22074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:27.574407 22074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:27.580186 22074 catalog_manager.cc:1347] Generated new cluster ID: a2c64dc01e49452a815ab59724990b53
I20260925 22:53:27.580282 22074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:27.621618 22074 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:27.622747 22074 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:27.628079 22074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: Generated new TSK 0
I20260925 22:53:27.628816 22074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:53:27.660008 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:27.798234 22079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:27.798496 22079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:27.798543 22079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:27.803103 22079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:27.803167 22079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:27.803231 22079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:27.809042 22079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:27.810079 22079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:27.811116 22079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:27.820658 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:27.820652 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:27.820652 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:27.820838 22079 server_base.cc:1034] running on GCE node
I20260925 22:53:27.821290 22079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:27.821899 22079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:27.823088 22079 hybrid_clock.cc:648] HybridClock initialized: now 1790376807823078 us; error 31 us; skew 500 ppm
Sep 25 22:53:27 dist-test-slave-kb1g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376807, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:27.830523 22079 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:53:27.831421 22079 webserver.cc:460] Webserver started at http://127.4.219.129:43103/ using document root <none> and password file <none>
I20260925 22:53:27.832011 22079 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:27.832105 22079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:27.832343 22079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:27.834488 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "b0b933b290824d5b85c73ff7ddc54605"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "2f7c39bd59ad2e1728690aa303f57cbc"
server_key_iv: "ad6c84bf1057990e2ca0aab43431fd60"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.835048 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "b0b933b290824d5b85c73ff7ddc54605"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "2f7c39bd59ad2e1728690aa303f57cbc"
server_key_iv: "ad6c84bf1057990e2ca0aab43431fd60"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.838765 22079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:53:27.841135 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:27.842427 22079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:27.842583 22079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "b0b933b290824d5b85c73ff7ddc54605"
format_stamp: "Formatted at 2026-09-25 22:53:27 on dist-test-slave-kb1g"
server_key: "2f7c39bd59ad2e1728690aa303f57cbc"
server_key_iv: "ad6c84bf1057990e2ca0aab43431fd60"
server_key_version: "encryptionkey@0"
I20260925 22:53:27.842698 22079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:27.861065 22079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:27.864635 22079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:27.864837 22079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:27.865437 22079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:27.866267 22079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:27.866338 22079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:27.866410 22079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:27.866451 22079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:27.875742 22079 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:33847
I20260925 22:53:27.875767 22207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:33847 every 8 connection(s)
I20260925 22:53:27.876669 22079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:27.877254 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22079
I20260925 22:53:27.877396 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:27.877681 4974 external_mini_cluster.cc:1428] Setting key 055613977387043d0243208929df5696
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Sep 25 22:53:27 dist-test-slave-kb1g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376807, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:53:27.892637 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:27.878967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:45281 (local address 127.4.219.190:46239)
0925 22:53:27.879100 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:27.879103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:27.879636 (+ 533us) server_negotiation.cc:407] Connection header received
0925 22:53:27.881318 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:27.881321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:27.881367 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:27.881439 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:27.883060 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.883695 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.885608 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.885838 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.885917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:27.888723 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:27.888726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:27.888729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:27.888754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:27.890293 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:27.890906 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:27.890913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:27.890919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:27.891027 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:27.891359 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:27.891364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:27.891366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:27.891672 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:27.891781 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:27.892272 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 22:53:27.892401 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:53:27.893494 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:27.878966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46239 (local address 127.4.219.129:45281)
0925 22:53:27.879485 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:27.879541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:27.881079 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:27.881629 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:27.881641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:27.882179 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:27.882882 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:27.882906 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.883834 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.883838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:27.885492 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:27.885503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:27.885996 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:27.886003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:27.886215 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:27.886840 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:27.886865 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:27.888517 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:27.890410 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:27.890419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:27.890438 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:27.890779 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:27.891124 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:27.891127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:27.891128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:27.891253 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:27.891783 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:27.891788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:27.892013 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 22:53:27.892205 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 22:53:27.892524 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":169,"threads_started":1}
I20260925 22:53:27.894748 22208 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46239
I20260925 22:53:27.894999 22208 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:27.895483 22208 heartbeater.cc:511] Master 127.4.219.190:46239 requested a full tablet report, sending...
I20260925 22:53:27.896952 22017 ts_manager.cc:194] Registered new tserver with Master: b0b933b290824d5b85c73ff7ddc54605 (127.4.219.129:33847)
I20260925 22:53:27.898034 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:45281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:53:27.906157 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:28.019953 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:28.020231 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:28.020282 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:28.025112 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:28.025173 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:28.025239 22213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:28.030941 22213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:28.032063 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:28.033169 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:28.042829 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:28.043007 22213 server_base.cc:1034] running on GCE node
W20260925 22:53:28.042793 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:28.042793 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:28.043491 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:28.044113 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:28.045276 22213 hybrid_clock.cc:648] HybridClock initialized: now 1790376808045258 us; error 28 us; skew 500 ppm
Sep 25 22:53:28 dist-test-slave-kb1g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376808, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:28.048336 22213 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:53:28.049391 22213 webserver.cc:460] Webserver started at http://127.4.219.130:37837/ using document root <none> and password file <none>
I20260925 22:53:28.050001 22213 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:28.050107 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:28.050351 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:28.052747 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "292eb9a4fbe14acaa4c3de3f785c26fd"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "90f093fb57203e4b417b2bbebe90422c"
server_key_iv: "7ef1824404e04989fc8b538719cade00"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.053344 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "292eb9a4fbe14acaa4c3de3f785c26fd"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "90f093fb57203e4b417b2bbebe90422c"
server_key_iv: "7ef1824404e04989fc8b538719cade00"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.057256 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:53:28.059813 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.061168 22213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:28.061321 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "292eb9a4fbe14acaa4c3de3f785c26fd"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "90f093fb57203e4b417b2bbebe90422c"
server_key_iv: "7ef1824404e04989fc8b538719cade00"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.061453 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:28.078116 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:28.081178 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:28.081383 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:28.081995 22213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:28.082860 22213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:28.082930 22213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.082998 22213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:28.083045 22213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.091794 22213 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:42669
I20260925 22:53:28.091816 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:42669 every 8 connection(s)
I20260925 22:53:28.092797 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
Sep 25 22:53:28 dist-test-slave-kb1g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376808, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:28.102728 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22213
I20260925 22:53:28.102856 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:28.103253 4974 external_mini_cluster.cc:1428] Setting key badab9d17d0a14616b51019494ba6806
I20260925 22:53:28.106334 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.094628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:42373 (local address 127.4.219.190:46239)
0925 22:53:28.094768 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.094772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.095577 (+ 805us) server_negotiation.cc:407] Connection header received
0925 22:53:28.096462 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.096465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.096523 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.096648 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:28.097821 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.098354 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.099499 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.099728 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.099840 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.102350 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:28.102353 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:28.102355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:28.102379 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:28.104093 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.104686 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.104689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.104691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.104786 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.105119 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.105125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.105128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.105389 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:28.105493 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.105949 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.106088 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:53:28.107079 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.095001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46239 (local address 127.4.219.130:42373)
0925 22:53:28.095423 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.095463 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.096259 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.096822 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.096828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.097179 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:28.097673 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.097687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.098480 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.098484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.099372 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.099381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.099877 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.099881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.100053 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.100737 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:28.100754 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:28.102182 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:28.104220 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:28.104226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:28.104241 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:28.104567 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:28.104879 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:28.104881 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:28.104883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:28.105020 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:28.105490 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:28.105495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:28.105719 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.105870 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.106145 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":115,"threads_started":1}
I20260925 22:53:28.108307 22342 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46239
I20260925 22:53:28.108578 22342 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:28.109050 22342 heartbeater.cc:511] Master 127.4.219.190:46239 requested a full tablet report, sending...
I20260925 22:53:28.110082 22017 ts_manager.cc:194] Registered new tserver with Master: 292eb9a4fbe14acaa4c3de3f785c26fd (127.4.219.130:42669)
I20260925 22:53:28.110651 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:42373
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:53:28.130724 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:28.240231 22347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:28.240479 22347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:28.240521 22347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:28.244617 22347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:28.244669 22347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:28.244783 22347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:28.250133 22347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:46239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:28.251156 22347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:28.252161 22347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:28.261533 22353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:28.261550 22352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:28.261540 22355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:28.262086 22347 server_base.cc:1034] running on GCE node
I20260925 22:53:28.262444 22347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:28.262984 22347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:28.264178 22347 hybrid_clock.cc:648] HybridClock initialized: now 1790376808264141 us; error 50 us; skew 500 ppm
Sep 25 22:53:28 dist-test-slave-kb1g krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376808, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:28.266956 22347 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:53:28.267834 22347 webserver.cc:460] Webserver started at http://127.4.219.131:35617/ using document root <none> and password file <none>
I20260925 22:53:28.268332 22347 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:28.268388 22347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:28.268605 22347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:28.270603 22347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "96151e61ddf44be2a61dec697108c6b4"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "032b523aff1f0ab688eed2064bdc6198"
server_key_iv: "5219bf9535ace42f9b16e1264c296b35"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.271086 22347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "96151e61ddf44be2a61dec697108c6b4"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "032b523aff1f0ab688eed2064bdc6198"
server_key_iv: "5219bf9535ace42f9b16e1264c296b35"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.274753 22347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:53:28.277200 22362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.278460 22347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:53:28.278563 22347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "96151e61ddf44be2a61dec697108c6b4"
format_stamp: "Formatted at 2026-09-25 22:53:28 on dist-test-slave-kb1g"
server_key: "032b523aff1f0ab688eed2064bdc6198"
server_key_iv: "5219bf9535ace42f9b16e1264c296b35"
server_key_version: "encryptionkey@0"
I20260925 22:53:28.278649 22347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:28.304214 22347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:28.307432 22347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:28.307586 22347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:28.308130 22347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:28.308939 22347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:28.308985 22347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.309033 22347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:28.309049 22347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:28.318126 22347 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:38799
I20260925 22:53:28.318148 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:38799 every 8 connection(s)
I20260925 22:53:28.319042 22347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:28.327253 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22347
I20260925 22:53:28.327417 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:28.327786 4974 external_mini_cluster.cc:1428] Setting key 29017810d535209ca2c4f82c61f64bb2
Sep 25 22:53:28 dist-test-slave-kb1g krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376808, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:28.333401 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.320951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44681 (local address 127.4.219.190:46239)
0925 22:53:28.321118 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.321122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.321948 (+ 826us) server_negotiation.cc:407] Connection header received
0925 22:53:28.323336 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.323340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.323400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.323528 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:28.324719 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.325259 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.326402 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.326630 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.326741 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.329617 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:28.329621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:28.329624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:28.329658 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:28.331117 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.331791 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.331794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.331796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.331923 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.332335 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.332338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.332340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.332602 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:28.332680 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.333162 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.333265 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:53:28.334220 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.321348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:46239 (local address 127.4.219.131:44681)
0925 22:53:28.321799 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.321841 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.323120 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.323699 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.323706 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.324072 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:28.324574 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.324588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.325385 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.325390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.326274 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.326283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.326782 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.326786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.326984 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.327711 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:28.327735 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:28.329457 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:28.331247 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:28.331253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:28.331268 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:28.331656 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:28.332045 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:28.332048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:28.332049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:28.332228 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:28.332697 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:28.332702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:28.332942 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.333107 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.333368 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":126,"threads_started":1}
I20260925 22:53:28.335407 22476 heartbeater.cc:347] Connected to a master server at 127.4.219.190:46239
I20260925 22:53:28.335687 22476 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:28.336195 22476 heartbeater.cc:511] Master 127.4.219.190:46239 requested a full tablet report, sending...
I20260925 22:53:28.337281 22017 ts_manager.cc:194] Registered new tserver with Master: 96151e61ddf44be2a61dec697108c6b4 (127.4.219.131:38799)
I20260925 22:53:28.337860 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:44681
I20260925 22:53:28.341919 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:28.351123 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.343793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.4.219.190:46239)
0925 22:53:28.343927 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.343932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.344134 (+ 202us) server_negotiation.cc:407] Connection header received
0925 22:53:28.344198 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.344201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.344241 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.344308 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:28.345035 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.345542 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.346410 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.346630 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.346723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.348176 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:28.348179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:28.348181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:28.348203 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:28.349764 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.350135 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.350137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.350139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.350210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:28.350425 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:28.350427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:28.350428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:28.350634 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:28.350735 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.350895 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.350983 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:53:28.354079 22017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:28.357110 22017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:28.372032 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.366223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.4.219.130:42669)
0925 22:53:28.366711 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.366727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.366847 (+ 120us) server_negotiation.cc:407] Connection header received
0925 22:53:28.366927 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.366932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.367092 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.367252 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:28.367678 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.368991 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.370433 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371320 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371432 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.371575 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.371770 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.371882 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":143,"threads_started":1}
I20260925 22:53:28.372352 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.365566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.4.219.131:38799)
0925 22:53:28.366107 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.366161 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.366786 (+ 625us) server_negotiation.cc:407] Connection header received
0925 22:53:28.366872 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.366879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.367093 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.367253 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:28.367679 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.368738 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.370559 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371327 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371432 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.371576 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.372122 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.372225 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":136,"threads_started":1}
I20260925 22:53:28.372557 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.366263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.4.219.129:33847)
0925 22:53:28.366961 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.366974 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.367595 (+ 621us) server_negotiation.cc:407] Connection header received
0925 22:53:28.367762 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.367770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.367981 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.368147 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:28.368338 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.369647 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.370977 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371682 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.371902 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.372286 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.372383 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":120,"threads_started":1}
I20260925 22:53:28.372036 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.365532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:42669 (local address 127.0.0.1:33666)
0925 22:53:28.366713 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.366726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.366829 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.367264 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.367270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.367306 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:28.367528 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.367536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.369142 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.369145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.370312 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.370321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371452 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.371564 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.371606 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.371731 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.371851 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":129,"spinlock_wait_cycles":4864,"thread_start_us":91,"threads_started":1}
I20260925 22:53:28.372056 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.365425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:38799 (local address 127.0.0.1:55792)
0925 22:53:28.366496 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.366549 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.366688 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.367264 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.367270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.367307 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:28.367528 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.367536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.368897 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.368900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.370428 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.370439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371657 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.371760 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.371784 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.371855 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.371918 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":201,"spinlock_wait_cycles":9856,"thread_start_us":72,"threads_started":1}
I20260925 22:53:28.372961 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.365802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33847 (local address 127.0.0.1:56270)
0925 22:53:28.367486 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.367496 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.367610 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.368140 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.368143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.368161 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:28.368241 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.368247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.369783 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.369787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.370867 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.370874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.371960 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.371963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.372041 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.372075 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.372147 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.372214 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1632,"mutex_wait_us":79,"thread_start_us":89,"threads_started":1}
I20260925 22:53:28.375541 22142 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 (DEFAULT_TABLE table=test-table [id=598f908a918e40b6bb2390259cb09bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:28.375536 22410 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 (DEFAULT_TABLE table=test-table [id=598f908a918e40b6bb2390259cb09bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:28.376652 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea2ab2e407042cfab0a3f540c0301b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:28.376667 22410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea2ab2e407042cfab0a3f540c0301b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:28.383167 22493 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Bootstrap starting.
I20260925 22:53:28.382732 22276 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 (DEFAULT_TABLE table=test-table [id=598f908a918e40b6bb2390259cb09bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:28.383912 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea2ab2e407042cfab0a3f540c0301b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:28.384658 22494 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Bootstrap starting.
I20260925 22:53:28.385779 22493 tablet_bootstrap.cc:654] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:28.386420 22494 tablet_bootstrap.cc:654] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:28.387168 22493 log.cc:824] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:28.387605 22494 log.cc:824] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:28.389393 22493 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: No bootstrap required, opened a new log
I20260925 22:53:28.389564 22494 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: No bootstrap required, opened a new log
I20260925 22:53:28.389595 22493 ts_tablet_manager.cc:1397] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:53:28.389727 22494 ts_tablet_manager.cc:1397] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:53:28.391065 22497 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Bootstrap starting.
I20260925 22:53:28.393317 22497 tablet_bootstrap.cc:654] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:28.393519 22493 raft_consensus.cc:348] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.393929 22493 raft_consensus.cc:374] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:28.394052 22493 raft_consensus.cc:729] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b933b290824d5b85c73ff7ddc54605, State: Initialized, Role: FOLLOWER
I20260925 22:53:28.393700 22494 raft_consensus.cc:348] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.394212 22494 raft_consensus.cc:374] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:28.394196 22497 log.cc:824] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:28.394304 22494 raft_consensus.cc:729] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96151e61ddf44be2a61dec697108c6b4, State: Initialized, Role: FOLLOWER
I20260925 22:53:28.394687 22493 consensus_queue.cc:260] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.394687 22494 consensus_queue.cc:260] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.395311 22208 heartbeater.cc:503] Master 127.4.219.190:46239 was elected leader, sending a full tablet report...
I20260925 22:53:28.395663 22494 ts_tablet_manager.cc:1428] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:53:28.395663 22493 ts_tablet_manager.cc:1428] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:53:28.396123 22497 tablet_bootstrap.cc:492] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: No bootstrap required, opened a new log
I20260925 22:53:28.396276 22476 heartbeater.cc:503] Master 127.4.219.190:46239 was elected leader, sending a full tablet report...
I20260925 22:53:28.396315 22497 ts_tablet_manager.cc:1397] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:53:28.400032 22497 raft_consensus.cc:348] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.400262 22497 raft_consensus.cc:374] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:28.400295 22497 raft_consensus.cc:729] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292eb9a4fbe14acaa4c3de3f785c26fd, State: Initialized, Role: FOLLOWER
I20260925 22:53:28.400710 22497 consensus_queue.cc:260] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.401279 22342 heartbeater.cc:503] Master 127.4.219.190:46239 was elected leader, sending a full tablet report...
I20260925 22:53:28.401511 22497 ts_tablet_manager.cc:1428] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260925 22:53:28.569994 22477 tablet.cc:2369] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:53:28.594005 22343 tablet.cc:2369] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:53:28.628011 22209 tablet.cc:2369] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:28.743723 22499 raft_consensus.cc:482] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:28.744017 22499 raft_consensus.cc:504] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.745222 22499 leader_election.cc:288] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96151e61ddf44be2a61dec697108c6b4 (127.4.219.131:38799), 292eb9a4fbe14acaa4c3de3f785c26fd (127.4.219.130:42669)
I20260925 22:53:28.750757 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.745581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:51845 (local address 127.4.219.131:38799)
0925 22:53:28.745727 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.745732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.746024 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:53:28.746136 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.746140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.746197 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.746302 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:28.746649 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.747545 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.749361 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.750056 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.750184 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.750239 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.750545 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.750621 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:53:28.750757 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.745625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:49939 (local address 127.4.219.130:42669)
0925 22:53:28.745842 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.745845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.746102 (+ 257us) server_negotiation.cc:407] Connection header received
0925 22:53:28.746161 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.746164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.746214 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.746310 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:28.746648 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.747545 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.749361 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.750048 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.750184 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.750239 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.750545 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.750621 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 22:53:28.750780 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.745511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:42669 (local address 127.4.219.129:49939)
0925 22:53:28.745979 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.745993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.746089 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.746321 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.746324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.746364 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:28.746521 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.746528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.747673 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.747677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.749221 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.749233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.750185 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.750189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.750359 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.750412 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.750485 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.750582 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":177,"spinlock_wait_cycles":4992,"thread_start_us":112,"threads_started":1}
I20260925 22:53:28.750788 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.745472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:38799 (local address 127.4.219.129:51845)
0925 22:53:28.745882 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.745898 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:28.746031 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:28.746321 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:28.746324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:28.746363 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:28.746526 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.746530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.747673 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.747677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.749221 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:28.749233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.750185 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.750189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:28.750363 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.750403 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:53:28.750486 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:53:28.750582 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":248,"spinlock_wait_cycles":27648,"thread_start_us":125,"threads_started":1}
I20260925 22:53:28.751595 22296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea2ab2e407042cfab0a3f540c0301b3" candidate_uuid: "b0b933b290824d5b85c73ff7ddc54605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" is_pre_election: true
I20260925 22:53:28.751626 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea2ab2e407042cfab0a3f540c0301b3" candidate_uuid: "b0b933b290824d5b85c73ff7ddc54605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96151e61ddf44be2a61dec697108c6b4" is_pre_election: true
I20260925 22:53:28.751902 22296 raft_consensus.cc:2454] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b933b290824d5b85c73ff7ddc54605 in term 0.
I20260925 22:53:28.751921 22430 raft_consensus.cc:2454] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b933b290824d5b85c73ff7ddc54605 in term 0.
I20260925 22:53:28.752415 22097 leader_election.cc:302] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292eb9a4fbe14acaa4c3de3f785c26fd, b0b933b290824d5b85c73ff7ddc54605; no voters:
I20260925 22:53:28.752728 22499 raft_consensus.cc:2781] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:28.752833 22499 raft_consensus.cc:482] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:28.752942 22499 raft_consensus.cc:3037] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:28.754045 22499 raft_consensus.cc:504] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.754487 22499 leader_election.cc:288] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [CANDIDATE]: Term 1 election: Requested vote from peers 96151e61ddf44be2a61dec697108c6b4 (127.4.219.131:38799), 292eb9a4fbe14acaa4c3de3f785c26fd (127.4.219.130:42669)
I20260925 22:53:28.754855 22430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea2ab2e407042cfab0a3f540c0301b3" candidate_uuid: "b0b933b290824d5b85c73ff7ddc54605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96151e61ddf44be2a61dec697108c6b4"
I20260925 22:53:28.754896 22296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea2ab2e407042cfab0a3f540c0301b3" candidate_uuid: "b0b933b290824d5b85c73ff7ddc54605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd"
I20260925 22:53:28.754999 22430 raft_consensus.cc:3037] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:28.755026 22296 raft_consensus.cc:3037] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:28.756115 22296 raft_consensus.cc:2454] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b933b290824d5b85c73ff7ddc54605 in term 1.
I20260925 22:53:28.756129 22430 raft_consensus.cc:2454] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b933b290824d5b85c73ff7ddc54605 in term 1.
I20260925 22:53:28.756453 22097 leader_election.cc:302] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292eb9a4fbe14acaa4c3de3f785c26fd, b0b933b290824d5b85c73ff7ddc54605; no voters:
I20260925 22:53:28.756675 22499 raft_consensus.cc:2781] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:28.756906 22499 raft_consensus.cc:686] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [term 1 LEADER]: Becoming Leader. State: Replica: b0b933b290824d5b85c73ff7ddc54605, State: Running, Role: LEADER
I20260925 22:53:28.757314 22499 consensus_queue.cc:237] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } }
I20260925 22:53:28.760746 22017 catalog_manager.cc:5495] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0b933b290824d5b85c73ff7ddc54605 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "b0b933b290824d5b85c73ff7ddc54605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b933b290824d5b85c73ff7ddc54605" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 33847 } health_report { overall_health: HEALTHY } } }
I20260925 22:53:28.821928 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.818262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.4.219.129:33847)
0925 22:53:28.818440 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:28.818447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:28.818467 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:53:28.818580 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:28.818586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:28.818678 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:28.818806 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:28.819135 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.819725 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.820584 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:28.820800 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:28.820884 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:28.821141 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:28.821555 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:28.821642 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:28.821742 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 22:53:28.821791 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:53:28.827865 22430 raft_consensus.cc:1261] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 1 FOLLOWER]: Refusing update from remote peer b0b933b290824d5b85c73ff7ddc54605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:28.827860 22296 raft_consensus.cc:1261] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 1 FOLLOWER]: Refusing update from remote peer b0b933b290824d5b85c73ff7ddc54605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:28.828598 22499 consensus_queue.cc:1035] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96151e61ddf44be2a61dec697108c6b4" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 38799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:28.828801 22504 consensus_queue.cc:1035] T 8ea2ab2e407042cfab0a3f540c0301b3 P b0b933b290824d5b85c73ff7ddc54605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "292eb9a4fbe14acaa4c3de3f785c26fd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 42669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:28.835842 22507 mvcc.cc:205] Tried to move back new op lower bound from 7333383408951971840 to 7333383408672915456. Current Snapshot: MvccSnapshot[applied={T|T < 7333383408951971840}]
I20260925 22:53:28.837646 22509 mvcc.cc:205] Tried to move back new op lower bound from 7333383408951971840 to 7333383408672915456. Current Snapshot: MvccSnapshot[applied={T|T < 7333383408951971840}]
I20260925 22:53:28.843860 22016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:28.844040 22016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:28.846484 22016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ebaef1e8f354a75bd39affa0b760095: Sending DeleteTablet for 3 replicas of tablet 8ea2ab2e407042cfab0a3f540c0301b3
I20260925 22:53:28.847316 22410 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:55792
I20260925 22:53:28.847337 22276 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:33666
I20260925 22:53:28.847429 22142 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea2ab2e407042cfab0a3f540c0301b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:28 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:56270
I20260925 22:53:28.847931 22518 tablet_replica.cc:330] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: stopping tablet replica
I20260925 22:53:28.848050 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22079
I20260925 22:53:28.848263 22518 raft_consensus.cc:2229] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:28.848572 22518 raft_consensus.cc:2258] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:28.849853 22518 ts_tablet_manager.cc:1905] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:28.852178 22518 ts_tablet_manager.cc:1918] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:28.852259 22518 log.cc:1196] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/8ea2ab2e407042cfab0a3f540c0301b3
I20260925 22:53:28.852622 22518 ts_tablet_manager.cc:1939] T 8ea2ab2e407042cfab0a3f540c0301b3 P 292eb9a4fbe14acaa4c3de3f785c26fd: Deleting consensus metadata
I20260925 22:53:28.853086 22517 tablet_replica.cc:330] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: stopping tablet replica
I20260925 22:53:28.853417 22517 raft_consensus.cc:2229] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:28.853725 22517 raft_consensus.cc:2258] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:28.853881 22003 catalog_manager.cc:4841] TS 292eb9a4fbe14acaa4c3de3f785c26fd (127.4.219.130:42669): tablet 8ea2ab2e407042cfab0a3f540c0301b3 (table test-table [id=598f908a918e40b6bb2390259cb09bd5]) successfully deleted
I20260925 22:53:28.855051 22517 ts_tablet_manager.cc:1905] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:53:28.855412 22002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:33847 (error 108)
I20260925 22:53:28.856210 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22213
I20260925 22:53:28.856451 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:28.856107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:33847 (local address 127.0.0.1:56276)
0925 22:53:28.856262 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:28.856341 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:53:28.857622 22517 ts_tablet_manager.cc:1918] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:28.857686 22517 log.cc:1196] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/8ea2ab2e407042cfab0a3f540c0301b3
I20260925 22:53:28.857893 22517 ts_tablet_manager.cc:1939] T 8ea2ab2e407042cfab0a3f540c0301b3 P 96151e61ddf44be2a61dec697108c6b4: Deleting consensus metadata
W20260925 22:53:28.858232 22002 catalog_manager.cc:4584] TS b0b933b290824d5b85c73ff7ddc54605 (127.4.219.129:33847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ea2ab2e407042cfab0a3f540c0301b3: Network error: Client connection negotiation failed: client connection to 127.4.219.129:33847: connect: Connection refused (error 111)
I20260925 22:53:28.859156 22004 catalog_manager.cc:4841] TS 96151e61ddf44be2a61dec697108c6b4 (127.4.219.131:38799): tablet 8ea2ab2e407042cfab0a3f540c0301b3 (table test-table [id=598f908a918e40b6bb2390259cb09bd5]) successfully deleted
I20260925 22:53:28.862314 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22347
I20260925 22:53:28.867653 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 21985
2026-09-25T22:53:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2674 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:28Z Disabled control of system clock
I20260925 22:53:28.898059 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34743
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:29.015612 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:29.015910 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:29.015951 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:29.020510 22524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:29.020599 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:29.020617 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:29.020637 22524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:29.020653 22524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:29.026155 22524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:34743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:29.027311 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:29.028796 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:29.037993 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.037993 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.038085 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:29.038303 22524 server_base.cc:1034] running on GCE node
I20260925 22:53:29.038993 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:29.040165 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:29.041394 22524 hybrid_clock.cc:648] HybridClock initialized: now 1790376809041331 us; error 73 us; skew 500 ppm
I20260925 22:53:29.043437 22524 webserver.cc:460] Webserver started at http://127.4.219.190:32947/ using document root <none> and password file <none>
I20260925 22:53:29.044087 22524 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:29.044196 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:29.044447 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:29.046733 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "e4548da3651e43cd8e3a7cb0295c4e60"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "9954f84f69fff9a741281adebd391717"
server_key_iv: "cfb098c713a4daff6b268cfb2df07ff2"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.047422 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "e4548da3651e43cd8e3a7cb0295c4e60"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "9954f84f69fff9a741281adebd391717"
server_key_iv: "cfb098c713a4daff6b268cfb2df07ff2"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.051460 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:53:29.054051 22538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.055408 22524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:29.055619 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "e4548da3651e43cd8e3a7cb0295c4e60"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "9954f84f69fff9a741281adebd391717"
server_key_iv: "cfb098c713a4daff6b268cfb2df07ff2"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.055744 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:29.072516 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:29.073280 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:29.073495 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:29.080649 22524 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:34743
I20260925 22:53:29.080672 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:34743 every 8 connection(s)
I20260925 22:53:29.081673 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:29.084095 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22524
I20260925 22:53:29.084239 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:29.084589 4974 external_mini_cluster.cc:1428] Setting key b37ed26543d5d38d6b0230f497133d3d
I20260925 22:53:29.085057 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:29.091403 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: Bootstrap starting.
I20260925 22:53:29.093570 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:29.094396 22591 log.cc:824] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:29.094761 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.086126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.4.219.190:34743)
0925 22:53:29.086792 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.086807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.086845 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:53:29.087831 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.087851 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.088311 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.088712 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:29.089186 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.090224 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.091204 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.091781 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.091973 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.092964 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:29.092970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:29.092983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:29.093390 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:29.093448 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:29.093469 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:29.093592 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.093718 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.093985 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260925 22:53:29.096429 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: No bootstrap required, opened a new log
I20260925 22:53:29.099891 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } }
I20260925 22:53:29.100106 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:29.100142 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4548da3651e43cd8e3a7cb0295c4e60, State: Initialized, Role: FOLLOWER
I20260925 22:53:29.100476 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } }
I20260925 22:53:29.100704 22591 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:29.100771 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:29.100875 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:29.101795 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } }
I20260925 22:53:29.102133 22591 leader_election.cc:302] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4548da3651e43cd8e3a7cb0295c4e60; no voters:
I20260925 22:53:29.102463 22591 leader_election.cc:288] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:29.102562 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:29.102741 22596 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [term 1 LEADER]: Becoming Leader. State: Replica: e4548da3651e43cd8e3a7cb0295c4e60, State: Running, Role: LEADER
I20260925 22:53:29.103020 22596 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } }
I20260925 22:53:29.103682 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:29.104753 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } } }
I20260925 22:53:29.104887 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:29.105142 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4548da3651e43cd8e3a7cb0295c4e60. Latest consensus state: current_term: 1 leader_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4548da3651e43cd8e3a7cb0295c4e60" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 34743 } } }
I20260925 22:53:29.105237 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:29.105300 22605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:29.108116 22605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:29.113986 22605 catalog_manager.cc:1347] Generated new cluster ID: eb5cdcacff54421aa6883529fc942e24
I20260925 22:53:29.114071 22605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:29.123803 22605 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:29.124867 22605 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:29.128846 22605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: Generated new TSK 0
I20260925 22:53:29.129508 22605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:29.139962 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--builtin_ntp_servers=127.4.219.148:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:29.251447 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:29.251703 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:29.251742 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:29.255936 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:29.255990 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:29.256075 22615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:29.261035 22615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:29.262045 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:29.263391 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:29.272498 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.272522 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:29.272682 22615 server_base.cc:1034] running on GCE node
W20260925 22:53:29.272509 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:29.273198 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:29.273761 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:29.274914 22615 hybrid_clock.cc:648] HybridClock initialized: now 1790376809274880 us; error 49 us; skew 500 ppm
I20260925 22:53:29.276739 22615 webserver.cc:460] Webserver started at http://127.4.219.129:46021/ using document root <none> and password file <none>
I20260925 22:53:29.277240 22615 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:29.277297 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:29.277446 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:29.279357 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "c387f794cbbe4c5bbfdec883e564e98a"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "564eaf81dac5c8c8ee5f2489af26da83"
server_key_iv: "151d170e0bbbb40c1e1d1874375188fa"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.279851 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "c387f794cbbe4c5bbfdec883e564e98a"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "564eaf81dac5c8c8ee5f2489af26da83"
server_key_iv: "151d170e0bbbb40c1e1d1874375188fa"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.283336 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260925 22:53:29.286029 22629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.287721 22615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:53:29.287870 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "c387f794cbbe4c5bbfdec883e564e98a"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "564eaf81dac5c8c8ee5f2489af26da83"
server_key_iv: "151d170e0bbbb40c1e1d1874375188fa"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.288015 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:29.309599 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:29.310262 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:29.310421 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:29.310956 22615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:29.311842 22615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:29.311897 22615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.311945 22615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:29.311970 22615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.320385 22615 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:44677
I20260925 22:53:29.320452 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:44677 every 8 connection(s)
I20260925 22:53:29.321334 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:29.326658 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22615
I20260925 22:53:29.326782 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:29.327082 4974 external_mini_cluster.cc:1428] Setting key 7c6485abf0efe2e2c4750ea3850cf0a9
I20260925 22:53:29.329015 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--builtin_ntp_servers=127.4.219.148:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:29.330643 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.323103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:56493 (local address 127.4.219.190:34743)
0925 22:53:29.323240 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.323244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.324207 (+ 963us) server_negotiation.cc:407] Connection header received
0925 22:53:29.325614 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.325618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.325687 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.325774 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:29.326655 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.327455 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.328678 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.328914 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.329031 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.330155 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:29.330158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:29.330161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:29.330180 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:29.330200 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:29.330208 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:29.330283 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.330442 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.330505 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:53:29.331468 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.323464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34743 (local address 127.4.219.129:56493)
0925 22:53:29.324084 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.324124 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.325382 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.325932 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.325938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.326018 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:29.326485 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.326511 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.327590 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.327599 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.328470 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.328479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.329063 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.329067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.329282 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.329893 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:29.329895 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:29.329932 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:29.329990 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:29.330297 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:29.330305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:29.330312 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.330442 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.330718 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260925 22:53:29.332753 22743 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34743
I20260925 22:53:29.332983 22743 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:29.333478 22743 heartbeater.cc:511] Master 127.4.219.190:34743 requested a full tablet report, sending...
I20260925 22:53:29.334991 22555 ts_manager.cc:194] Registered new tserver with Master: c387f794cbbe4c5bbfdec883e564e98a (127.4.219.129:44677)
I20260925 22:53:29.336133 22555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.129:56493
W20260925 22:53:29.451781 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:29.452076 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:29.452157 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:29.456199 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:29.456281 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:29.456368 22746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:29.461668 22746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:29.462764 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:29.464155 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:29.473948 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.474022 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.474186 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:29.474088 22746 server_base.cc:1034] running on GCE node
I20260925 22:53:29.474617 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:29.475286 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:29.476472 22746 hybrid_clock.cc:648] HybridClock initialized: now 1790376809476436 us; error 45 us; skew 500 ppm
I20260925 22:53:29.478263 22746 webserver.cc:460] Webserver started at http://127.4.219.130:45091/ using document root <none> and password file <none>
I20260925 22:53:29.478868 22746 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:29.478967 22746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:29.479197 22746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:29.481341 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "09fa52d055a948eca9fee0444dca65ad"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "121c45593b635a792060e5a1dfd1e81a"
server_key_iv: "58340a684c44c45a62c50cf780794adf"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.481889 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "09fa52d055a948eca9fee0444dca65ad"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "121c45593b635a792060e5a1dfd1e81a"
server_key_iv: "58340a684c44c45a62c50cf780794adf"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.485668 22746 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260925 22:53:29.488083 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.489398 22746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:53:29.489539 22746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "09fa52d055a948eca9fee0444dca65ad"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "121c45593b635a792060e5a1dfd1e81a"
server_key_iv: "58340a684c44c45a62c50cf780794adf"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.489653 22746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:29.501467 22746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:29.502139 22746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:29.502336 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:29.502905 22746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:29.503948 22746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:29.504017 22746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.504088 22746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:29.504140 22746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.513242 22746 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:38475
I20260925 22:53:29.513299 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:38475 every 8 connection(s)
I20260925 22:53:29.514196 22746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:29.515669 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22746
I20260925 22:53:29.515799 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:29.516144 4974 external_mini_cluster.cc:1428] Setting key 38366f73114970530a4acf8bf5fbc230
I20260925 22:53:29.518385 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--builtin_ntp_servers=127.4.219.148:32923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:29.524741 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.516130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:43623 (local address 127.4.219.190:34743)
0925 22:53:29.516279 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.516284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.517562 (+ 1278us) server_negotiation.cc:407] Connection header received
0925 22:53:29.519138 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.519141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.519196 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.519276 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:29.520181 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.520788 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.522579 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.522799 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.522889 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.524212 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:29.524215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:29.524218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:29.524230 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:29.524249 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:29.524257 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:29.524315 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.524562 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.524628 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:53:29.526005 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.516687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34743 (local address 127.4.219.130:43623)
0925 22:53:29.517414 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.517454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.518907 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.519457 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.519464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.519551 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:29.520036 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.520048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.520921 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.520928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.522448 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.522459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.522965 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.522972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.523213 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.523887 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:29.523892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:29.523953 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:29.524046 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:29.524346 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:29.524364 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:29.524373 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.524519 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.524868 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260925 22:53:29.527226 22874 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34743
I20260925 22:53:29.527467 22874 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:29.527985 22874 heartbeater.cc:511] Master 127.4.219.190:34743 requested a full tablet report, sending...
I20260925 22:53:29.529132 22555 ts_manager.cc:194] Registered new tserver with Master: 09fa52d055a948eca9fee0444dca65ad (127.4.219.130:38475)
I20260925 22:53:29.530009 22555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.130:43623
W20260925 22:53:29.636976 22877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:29.637255 22877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:29.637297 22877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:29.641902 22877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:29.641973 22877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:29.642031 22877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:29.646828 22877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:32923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:34743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:29.647902 22877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:29.649338 22877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:29.659005 22883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.659035 22882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:29.659051 22885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:29.659394 22877 server_base.cc:1034] running on GCE node
I20260925 22:53:29.659850 22877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:29.660627 22877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:29.661839 22877 hybrid_clock.cc:648] HybridClock initialized: now 1790376809661818 us; error 34 us; skew 500 ppm
I20260925 22:53:29.663810 22877 webserver.cc:460] Webserver started at http://127.4.219.131:38561/ using document root <none> and password file <none>
I20260925 22:53:29.664451 22877 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:29.664575 22877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:29.664796 22877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:29.666944 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "b9f170d114734540843ca9d8bd622f14"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "77110aded87455f50a7a8adde418ede9"
server_key_iv: "e1c2412cc288045bb623ff5ad8061203"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.667526 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "b9f170d114734540843ca9d8bd622f14"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "77110aded87455f50a7a8adde418ede9"
server_key_iv: "e1c2412cc288045bb623ff5ad8061203"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.671393 22877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:53:29.673898 22891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.675297 22877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:29.675462 22877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "b9f170d114734540843ca9d8bd622f14"
format_stamp: "Formatted at 2026-09-25 22:53:29 on dist-test-slave-kb1g"
server_key: "77110aded87455f50a7a8adde418ede9"
server_key_iv: "e1c2412cc288045bb623ff5ad8061203"
server_key_version: "encryptionkey@0"
I20260925 22:53:29.675587 22877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:29.691809 22877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:29.692493 22877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:29.692746 22877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:29.693333 22877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:29.694363 22877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:29.694425 22877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.694464 22877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:29.694510 22877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:29.703218 22877 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:34293
I20260925 22:53:29.703238 23004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:34293 every 8 connection(s)
I20260925 22:53:29.704175 22877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:29.705327 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 22877
I20260925 22:53:29.705473 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:29.705816 4974 external_mini_cluster.cc:1428] Setting key 5d3b20f4f25e7fdf2050a0f7ce32c7c3
I20260925 22:53:29.713511 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.706203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:50367 (local address 127.4.219.190:34743)
0925 22:53:29.706411 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.706415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.707641 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 22:53:29.709182 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.709184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.709229 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.709296 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:29.710126 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.710659 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.711774 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.711983 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.712060 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.713091 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:29.713094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:29.713096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:29.713108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:29.713126 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:29.713134 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:29.713193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.713328 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.713378 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 22:53:29.714430 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.706754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:34743 (local address 127.4.219.131:50367)
0925 22:53:29.707497 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.707549 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.708990 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.709432 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.709438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.709522 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:29.709997 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.710009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.710773 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.710775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.711666 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.711675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.712108 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.712111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.712256 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.712828 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:29.712829 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:29.712869 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:29.712943 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:29.713208 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:29.713215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:29.713218 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.713312 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.713542 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260925 22:53:29.715534 23005 heartbeater.cc:347] Connected to a master server at 127.4.219.190:34743
I20260925 22:53:29.715732 23005 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:29.716221 23005 heartbeater.cc:511] Master 127.4.219.190:34743 requested a full tablet report, sending...
I20260925 22:53:29.717242 22555 ts_manager.cc:194] Registered new tserver with Master: b9f170d114734540843ca9d8bd622f14 (127.4.219.131:34293)
I20260925 22:53:29.717805 22555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.131:50367
I20260925 22:53:29.719719 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:29.725162 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.721737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.4.219.190:34743)
0925 22:53:29.721886 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.721890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.722061 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:53:29.722150 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.722152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.722190 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.722253 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:29.722471 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.722957 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.723774 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.723974 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.724048 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.724769 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:29.724772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:29.724774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:29.724785 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:29.724801 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:29.724809 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:29.724865 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.724939 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.724986 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:53:29.727632 22555 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:29.730567 22555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:29.744413 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.738535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:34293 (local address 127.0.0.1:52864)
0925 22:53:29.739063 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.739115 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.739245 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.740334 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.740337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.740348 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:29.740608 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.740617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.741658 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.741664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.742921 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.742938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.743844 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.743848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.743950 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.743989 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.744109 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.744234 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260925 22:53:29.744446 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.738692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.4.219.131:34293)
0925 22:53:29.739931 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.739944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.739973 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:53:29.740037 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.740040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.740178 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.740479 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:29.740716 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.741534 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.743050 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.743710 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.743817 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.744005 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.744158 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.744274 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":177,"threads_started":1}
I20260925 22:53:29.744736 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.738666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:44677 (local address 127.0.0.1:33676)
0925 22:53:29.739481 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.739494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.739576 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.740301 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.740309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.740335 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:29.740613 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.740622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.742079 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.742085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.743267 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.743278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.744326 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.744330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.744441 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.744476 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.744550 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.744614 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":75,"thread_start_us":88,"threads_started":1}
I20260925 22:53:29.744969 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.739269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38475 (local address 127.0.0.1:37468)
0925 22:53:29.739752 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.739766 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.739845 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.740716 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.740718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.740735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:29.740818 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.740823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.742160 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.742163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.743305 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.743315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.744376 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.744380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.744643 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.744667 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.744733 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.744818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":32,"thread_start_us":85,"threads_started":1}
I20260925 22:53:29.745869 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.739373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.4.219.129:44677)
0925 22:53:29.739781 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.739794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.739819 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:53:29.739900 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.739907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.740122 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.740279 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:29.740751 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.741930 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.743404 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.744192 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.745437 (+ 1245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.745626 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.745682 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.745753 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":100,"threads_started":1}
I20260925 22:53:29.747268 22939 tablet_service.cc:1467] Processing CreateTablet for tablet 493121592a344e628462c45ba7c2f168 (DEFAULT_TABLE table=test-table [id=04382d970ca6494b8ec9a0bb1cf790f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:29.748200 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493121592a344e628462c45ba7c2f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:29.748035 22677 tablet_service.cc:1467] Processing CreateTablet for tablet 493121592a344e628462c45ba7c2f168 (DEFAULT_TABLE table=test-table [id=04382d970ca6494b8ec9a0bb1cf790f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:29.749042 22677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493121592a344e628462c45ba7c2f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:29.752640 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.739662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.4.219.130:38475)
0925 22:53:29.740034 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.740045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.740065 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:53:29.740131 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.740138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.740296 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.740420 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:29.740943 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.742049 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.743433 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.744063 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.744154 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.744315 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.745139 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.745238 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":99,"threads_started":1}
I20260925 22:53:29.754647 23022 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Bootstrap starting.
I20260925 22:53:29.755264 23023 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Bootstrap starting.
I20260925 22:53:29.755439 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 493121592a344e628462c45ba7c2f168 (DEFAULT_TABLE table=test-table [id=04382d970ca6494b8ec9a0bb1cf790f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:29.756240 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493121592a344e628462c45ba7c2f168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:29.756903 23022 tablet_bootstrap.cc:654] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:29.757309 23023 tablet_bootstrap.cc:654] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:29.757743 23022 log.cc:824] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:29.758065 23023 log.cc:824] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:29.759795 23022 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: No bootstrap required, opened a new log
I20260925 22:53:29.759977 23022 ts_tablet_manager.cc:1397] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:53:29.760638 23023 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: No bootstrap required, opened a new log
I20260925 22:53:29.760843 23023 ts_tablet_manager.cc:1397] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:53:29.762655 23026 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Bootstrap starting.
I20260925 22:53:29.764005 23022 raft_consensus.cc:348] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.764230 23022 raft_consensus.cc:374] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:29.764283 23022 raft_consensus.cc:729] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f170d114734540843ca9d8bd622f14, State: Initialized, Role: FOLLOWER
I20260925 22:53:29.764381 23023 raft_consensus.cc:348] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.764655 23023 raft_consensus.cc:374] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:29.764736 23026 tablet_bootstrap.cc:654] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:29.764734 23023 raft_consensus.cc:729] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c387f794cbbe4c5bbfdec883e564e98a, State: Initialized, Role: FOLLOWER
I20260925 22:53:29.764782 23022 consensus_queue.cc:260] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.765196 23023 consensus_queue.cc:260] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.765573 23005 heartbeater.cc:503] Master 127.4.219.190:34743 was elected leader, sending a full tablet report...
I20260925 22:53:29.765640 23026 log.cc:824] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:29.765782 23022 ts_tablet_manager.cc:1428] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:53:29.766134 23023 ts_tablet_manager.cc:1428] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:53:29.766228 22743 heartbeater.cc:503] Master 127.4.219.190:34743 was elected leader, sending a full tablet report...
I20260925 22:53:29.768141 23026 tablet_bootstrap.cc:492] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: No bootstrap required, opened a new log
I20260925 22:53:29.768337 23026 ts_tablet_manager.cc:1397] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 22:53:29.771664 23026 raft_consensus.cc:348] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.771852 23026 raft_consensus.cc:374] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:29.771924 23026 raft_consensus.cc:729] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fa52d055a948eca9fee0444dca65ad, State: Initialized, Role: FOLLOWER
I20260925 22:53:29.772286 23026 consensus_queue.cc:260] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.772886 22874 heartbeater.cc:503] Master 127.4.219.190:34743 was elected leader, sending a full tablet report...
I20260925 22:53:29.773183 23026 ts_tablet_manager.cc:1428] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260925 22:53:29.822540 22744 tablet.cc:2369] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:29.886464 23030 raft_consensus.cc:482] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:29.886713 23030 raft_consensus.cc:504] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.887918 23030 leader_election.cc:288] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9f170d114734540843ca9d8bd622f14 (127.4.219.131:34293), c387f794cbbe4c5bbfdec883e564e98a (127.4.219.129:44677)
I20260925 22:53:29.892416 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:44677 (local address 127.4.219.130:43437)
0925 22:53:29.888308 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.888325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.888443 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.888774 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.888777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.888803 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:29.888962 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.888973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.889933 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.889936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.891215 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.891225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.892016 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.892019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.892099 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.892119 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.892179 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.892233 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1,"spinlock_wait_cycles":6400}
I20260925 22:53:29.892468 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.888291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:43437 (local address 127.4.219.129:44677)
0925 22:53:29.888417 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.888422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.888436 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:53:29.888593 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.888598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.888653 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.888718 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:29.889184 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.889818 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.891332 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.891894 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.892017 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.892067 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.892222 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.892302 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:53:29.892910 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.888287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:56939 (local address 127.4.219.131:34293)
0925 22:53:29.888430 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.888437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.888793 (+ 356us) server_negotiation.cc:407] Connection header received
0925 22:53:29.888921 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.888923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.888987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.889179 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:29.889627 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.890192 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.891541 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.892186 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.892299 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.892338 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.892652 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.892730 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:53:29.893097 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493121592a344e628462c45ba7c2f168" candidate_uuid: "09fa52d055a948eca9fee0444dca65ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387f794cbbe4c5bbfdec883e564e98a" is_pre_election: true
I20260925 22:53:29.893330 23031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.888161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:34293 (local address 127.4.219.130:56939)
0925 22:53:29.888670 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.888681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:29.888801 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:29.889360 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:29.889363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:29.889388 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:29.889504 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.889509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.890292 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.890296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.891440 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:29.891447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.892303 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.892307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:29.892419 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.892486 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 22:53:29.893078 (+ 592us) client_negotiation.cc:240] Negotiation successful
0925 22:53:29.893156 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":153,"spinlock_wait_cycles":384,"thread_start_us":139,"threads_started":1}
I20260925 22:53:29.893455 22697 raft_consensus.cc:2454] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09fa52d055a948eca9fee0444dca65ad in term 0.
I20260925 22:53:29.893983 22959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493121592a344e628462c45ba7c2f168" candidate_uuid: "09fa52d055a948eca9fee0444dca65ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f170d114734540843ca9d8bd622f14" is_pre_election: true
I20260925 22:53:29.894032 22763 leader_election.cc:302] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fa52d055a948eca9fee0444dca65ad, c387f794cbbe4c5bbfdec883e564e98a; no voters:
I20260925 22:53:29.894279 22959 raft_consensus.cc:2454] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09fa52d055a948eca9fee0444dca65ad in term 0.
I20260925 22:53:29.894337 23030 raft_consensus.cc:2781] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:29.894428 23030 raft_consensus.cc:482] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:29.894480 23030 raft_consensus.cc:3037] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:29.895609 23030 raft_consensus.cc:504] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.896006 23030 leader_election.cc:288] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [CANDIDATE]: Term 1 election: Requested vote from peers b9f170d114734540843ca9d8bd622f14 (127.4.219.131:34293), c387f794cbbe4c5bbfdec883e564e98a (127.4.219.129:44677)
I20260925 22:53:29.896333 22959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493121592a344e628462c45ba7c2f168" candidate_uuid: "09fa52d055a948eca9fee0444dca65ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f170d114734540843ca9d8bd622f14"
I20260925 22:53:29.896392 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493121592a344e628462c45ba7c2f168" candidate_uuid: "09fa52d055a948eca9fee0444dca65ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c387f794cbbe4c5bbfdec883e564e98a"
I20260925 22:53:29.896457 22959 raft_consensus.cc:3037] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:29.896509 22697 raft_consensus.cc:3037] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:29.897763 22959 raft_consensus.cc:2454] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09fa52d055a948eca9fee0444dca65ad in term 1.
I20260925 22:53:29.897763 22697 raft_consensus.cc:2454] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09fa52d055a948eca9fee0444dca65ad in term 1.
I20260925 22:53:29.898078 22761 leader_election.cc:302] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fa52d055a948eca9fee0444dca65ad, b9f170d114734540843ca9d8bd622f14; no voters:
I20260925 22:53:29.898279 23030 raft_consensus.cc:2781] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:29.898535 23030 raft_consensus.cc:686] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 1 LEADER]: Becoming Leader. State: Replica: 09fa52d055a948eca9fee0444dca65ad, State: Running, Role: LEADER
I20260925 22:53:29.898900 23030 consensus_queue.cc:237] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } }
I20260925 22:53:29.902007 22555 catalog_manager.cc:5495] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 09fa52d055a948eca9fee0444dca65ad (127.4.219.130). New cstate: current_term: 1 leader_uuid: "09fa52d055a948eca9fee0444dca65ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09fa52d055a948eca9fee0444dca65ad" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 38475 } health_report { overall_health: HEALTHY } } }
I20260925 22:53:29.931196 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.928082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.4.219.130:38475)
0925 22:53:29.928213 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:29.928217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:29.928231 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:53:29.928308 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:29.928310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:29.928365 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:29.928454 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:29.928684 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.929170 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.929949 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:29.930148 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:29.930220 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:29.930465 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:29.930892 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:29.930979 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:29.931043 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:53:29.931087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:53:29.937216 22959 raft_consensus.cc:1261] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 1 FOLLOWER]: Refusing update from remote peer 09fa52d055a948eca9fee0444dca65ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:29.937216 22697 raft_consensus.cc:1261] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a [term 1 FOLLOWER]: Refusing update from remote peer 09fa52d055a948eca9fee0444dca65ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:29.938006 23032 consensus_queue.cc:1035] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c387f794cbbe4c5bbfdec883e564e98a" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:29.938205 23030 consensus_queue.cc:1035] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f170d114734540843ca9d8bd622f14" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:29.955965 22553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:29.956125 22553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:29.958562 22553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4548da3651e43cd8e3a7cb0295c4e60: Sending DeleteTablet for 3 replicas of tablet 493121592a344e628462c45ba7c2f168
I20260925 22:53:29.959424 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet 493121592a344e628462c45ba7c2f168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:29 UTC) from {username='slave'} at 127.0.0.1:37468
I20260925 22:53:29.959420 22939 tablet_service.cc:1514] Processing DeleteTablet for tablet 493121592a344e628462c45ba7c2f168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:29 UTC) from {username='slave'} at 127.0.0.1:52864
I20260925 22:53:29.959514 22677 tablet_service.cc:1514] Processing DeleteTablet for tablet 493121592a344e628462c45ba7c2f168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:29 UTC) from {username='slave'} at 127.0.0.1:33676
I20260925 22:53:29.960072 23046 tablet_replica.cc:330] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: stopping tablet replica
I20260925 22:53:29.960381 23045 tablet_replica.cc:330] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: stopping tablet replica
I20260925 22:53:29.960438 23047 tablet_replica.cc:330] T 493121592a344e628462c45ba7c2f168 P c387f794cbbe4c5bbfdec883e564e98a: stopping tablet replica
I20260925 22:53:29.960786 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22615
I20260925 22:53:29.960705 23045 raft_consensus.cc:2229] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:29.962023 23046 raft_consensus.cc:2229] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:53:29.962320 23045 raft_consensus.cc:2258] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:29.962404 23046 raft_consensus.cc:2258] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:29.963737 23045 ts_tablet_manager.cc:1905] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:29.966526 23045 ts_tablet_manager.cc:1918] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:29.966620 23045 log.cc:1196] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/493121592a344e628462c45ba7c2f168
I20260925 22:53:29.966925 23045 ts_tablet_manager.cc:1939] T 493121592a344e628462c45ba7c2f168 P b9f170d114734540843ca9d8bd622f14: Deleting consensus metadata
W20260925 22:53:29.967095 22541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:44677 (error 108)
I20260925 22:53:29.967852 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22746
I20260925 22:53:29.968191 23046 ts_tablet_manager.cc:1905] T 493121592a344e628462c45ba7c2f168 P 09fa52d055a948eca9fee0444dca65ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:29.968626 22539 catalog_manager.cc:4841] TS b9f170d114734540843ca9d8bd622f14 (127.4.219.131:34293): tablet 493121592a344e628462c45ba7c2f168 (table test-table [id=04382d970ca6494b8ec9a0bb1cf790f5]) successfully deleted
I20260925 22:53:29.969134 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.968832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:44677 (local address 127.0.0.1:33678)
0925 22:53:29.968943 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.969034 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:44677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 22:53:29.969826 22541 catalog_manager.cc:4584] TS c387f794cbbe4c5bbfdec883e564e98a (127.4.219.129:44677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 493121592a344e628462c45ba7c2f168: Network error: Client connection negotiation failed: client connection to 127.4.219.129:44677: connect: Connection refused (error 111)
I20260925 22:53:29.974088 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22877
I20260925 22:53:29.974601 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:29.974273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:38475 (local address 127.0.0.1:37488)
0925 22:53:29.974455 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:29.974507 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.130:38475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 22:53:29.979525 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 22524
2026-09-25T22:53:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1110 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:30 dist-test-slave-kb1g krb5kdc[23051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:30 dist-test-slave-kb1g krb5kdc[23051](info): set up 2 sockets
Sep 25 22:53:30 dist-test-slave-kb1g krb5kdc[23051](info): commencing operation
krb5kdc: starting...
W20260925 22:53:31.016407 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:31.133472 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:43821
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:31.243820 23064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:31.244081 23064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:31.244153 23064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:31.248158 23064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:31.248214 23064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:31.248234 23064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:31.248286 23064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:31.248319 23064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:31.253594 23064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:31.254659 23064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:31.255717 23064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:31.264253 23072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.264430 23064 server_base.cc:1034] running on GCE node
W20260925 22:53:31.264219 23070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.264219 23069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.265094 23064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:31.266161 23064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:31.267351 23064 hybrid_clock.cc:648] HybridClock initialized: now 1790376811267328 us; error 35 us; skew 500 ppm
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:31.270115 23064 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:31.270900 23064 webserver.cc:460] Webserver started at http://127.4.219.190:39027/ using document root <none> and password file <none>
I20260925 22:53:31.271396 23064 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:31.271453 23064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:31.271610 23064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:31.273617 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "314a26d561c0d2b63462b5d7c1e6235c"
server_key_iv: "191b1a55e8e75f781352b8375a136c84"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.274127 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "314a26d561c0d2b63462b5d7c1e6235c"
server_key_iv: "191b1a55e8e75f781352b8375a136c84"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.277709 23064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:53:31.280048 23079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.281279 23064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:31.281404 23064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "314a26d561c0d2b63462b5d7c1e6235c"
server_key_iv: "191b1a55e8e75f781352b8375a136c84"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.281509 23064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:31.304291 23064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:31.307200 23064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:31.307467 23064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:31.315052 23064 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:43821
I20260925 22:53:31.315056 23131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:43821 every 8 connection(s)
I20260925 22:53:31.316025 23064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:31.318878 23132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:31.319792 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23064
I20260925 22:53:31.319947 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:31.320237 4974 external_mini_cluster.cc:1428] Setting key 1b600cff4beaf89c1e489ffdebcc0976
I20260925 22:53:31.325068 23132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: Bootstrap starting.
I20260925 22:53:31.327327 23132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:31.328064 23132 log.cc:824] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:31.330189 23132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: No bootstrap required, opened a new log
I20260925 22:53:31.333880 23132 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } }
I20260925 22:53:31.334077 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:31.321652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.4.219.190:43821)
0925 22:53:31.322095 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:31.322109 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:31.322146 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:53:31.322802 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:31.322820 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:31.323153 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:31.323490 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:31.324284 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.325366 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.326230 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.326696 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.326856 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:31.329553 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:31.329558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:31.329571 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:31.329597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:31.331313 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.331760 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.331766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.331770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.331880 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.332157 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.332162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.332164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.332617 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:31.332824 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:31.333066 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 22:53:31.333352 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260925 22:53:31.334120 23132 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:31.334183 23132 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f430e48dfb4ef1b08ca9ad4e13c0f7, State: Initialized, Role: FOLLOWER
I20260925 22:53:31.334676 23132 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } }
I20260925 22:53:31.334812 23132 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:31.334863 23132 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:31.334942 23132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:31.335835 23132 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } }
I20260925 22:53:31.336197 23132 leader_election.cc:302] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f430e48dfb4ef1b08ca9ad4e13c0f7; no voters:
I20260925 22:53:31.336577 23132 leader_election.cc:288] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:31.336697 23137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:31.336925 23137 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [term 1 LEADER]: Becoming Leader. State: Replica: e5f430e48dfb4ef1b08ca9ad4e13c0f7, State: Running, Role: LEADER
I20260925 22:53:31.337220 23137 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } }
I20260925 22:53:31.337858 23132 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:31.338915 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } } }
I20260925 22:53:31.339051 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:31.339411 23146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:31.340112 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f430e48dfb4ef1b08ca9ad4e13c0f7. Latest consensus state: current_term: 1 leader_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f430e48dfb4ef1b08ca9ad4e13c0f7" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43821 } } }
I20260925 22:53:31.340224 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:31.342615 23146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:31.348264 23146 catalog_manager.cc:1347] Generated new cluster ID: 4999525e985b4466871e027b9f245994
I20260925 22:53:31.348332 23146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:31.361706 23146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:31.363056 23146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:31.367499 23146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: Generated new TSK 0
I20260925 22:53:31.368240 23146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:53:31.397311 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:31.510303 23158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:31.510560 23158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:31.510603 23158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:31.514751 23158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:31.514808 23158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:31.514899 23158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:31.520185 23158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:31.521291 23158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:31.522331 23158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:31.531930 23166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.531986 23158 server_base.cc:1034] running on GCE node
W20260925 22:53:31.531916 23164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.531915 23163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.532637 23158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:31.533267 23158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:31.534427 23158 hybrid_clock.cc:648] HybridClock initialized: now 1790376811534409 us; error 28 us; skew 500 ppm
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:31.537153 23158 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:53:31.538041 23158 webserver.cc:460] Webserver started at http://127.4.219.129:43723/ using document root <none> and password file <none>
I20260925 22:53:31.538689 23158 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:31.538781 23158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:31.539006 23158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:31.541132 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "173e52b315bbd218259e2ece1c2e6f5c"
server_key_iv: "bfbc660bef4eee34a01411c8586444bb"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.541668 23158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "173e52b315bbd218259e2ece1c2e6f5c"
server_key_iv: "bfbc660bef4eee34a01411c8586444bb"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.545181 23158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:53:31.547490 23173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.548687 23158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:31.548818 23158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "173e52b315bbd218259e2ece1c2e6f5c"
server_key_iv: "bfbc660bef4eee34a01411c8586444bb"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.548924 23158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:31.562351 23158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:31.565073 23158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:31.565264 23158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:31.565796 23158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:31.566572 23158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:31.566639 23158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.566708 23158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:31.566751 23158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.575013 23158 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:39845
I20260925 22:53:31.575043 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:39845 every 8 connection(s)
I20260925 22:53:31.575815 23158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:31.583781 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23158
I20260925 22:53:31.583930 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:31.584260 4974 external_mini_cluster.cc:1428] Setting key 3d1478993f91f8320fb404e436044576
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:31.588987 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:31.577528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:60607 (local address 127.4.219.190:43821)
0925 22:53:31.577692 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:31.577697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:31.578454 (+ 757us) server_negotiation.cc:407] Connection header received
0925 22:53:31.579295 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:31.579298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:31.579346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:31.579418 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:31.580523 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.581089 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.582290 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.582503 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.582575 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:31.585128 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:31.585153 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:31.585157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:31.585188 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:31.586740 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.587384 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.587389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.587393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.587499 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.587809 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.587814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.587818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.588062 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:31.588167 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:31.588619 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 22:53:31.588753 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:53:31.589694 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:31.577877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43821 (local address 127.4.219.129:60607)
0925 22:53:31.578305 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:31.578346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:31.579102 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:31.579590 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:31.579596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:31.579922 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:31.580364 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:31.580377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.581209 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.581214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:31.582187 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:31.582197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.582622 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.582625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:31.582760 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:31.583252 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:31.583266 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:31.584924 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:31.586857 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:31.586866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:31.586883 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:31.587268 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:31.587591 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:31.587594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:31.587595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:31.587718 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:31.588160 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:31.588165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:31.588373 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 22:53:31.588523 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 22:53:31.588815 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260925 22:53:31.590862 23287 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43821
I20260925 22:53:31.591102 23287 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:31.591562 23287 heartbeater.cc:511] Master 127.4.219.190:43821 requested a full tablet report, sending...
I20260925 22:53:31.593039 23096 ts_manager.cc:194] Registered new tserver with Master: 1d92c2c9c1e649fbba08a9fc2410ebc6 (127.4.219.129:39845)
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
I20260925 22:53:31.594198 23096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:60607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:53:31.609658 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:31.722846 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:31.723138 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:31.723182 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:31.727283 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:31.727350 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:31.727412 23292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:31.733256 23292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:31.734316 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:31.735307 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:31.745417 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.745445 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.745417 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.746383 23292 server_base.cc:1034] running on GCE node
I20260925 22:53:31.746989 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:31.747699 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:31.748947 23292 hybrid_clock.cc:648] HybridClock initialized: now 1790376811748913 us; error 48 us; skew 500 ppm
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:31.752086 23292 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:53:31.753304 23292 webserver.cc:460] Webserver started at http://127.4.219.130:36497/ using document root <none> and password file <none>
I20260925 22:53:31.753896 23292 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:31.753958 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:31.754133 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:31.756307 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db01f2f3dbee0579183c4ac9c0d673ee"
server_key_iv: "43ad93485cb2fe6f517b1061aaf4a88c"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.756919 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db01f2f3dbee0579183c4ac9c0d673ee"
server_key_iv: "43ad93485cb2fe6f517b1061aaf4a88c"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.760851 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:53:31.763337 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.764765 23292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:53:31.764868 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db01f2f3dbee0579183c4ac9c0d673ee"
server_key_iv: "43ad93485cb2fe6f517b1061aaf4a88c"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.764961 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:31.778326 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:31.781709 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:31.781862 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:31.782379 23292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:31.783174 23292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:31.783222 23292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.783262 23292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:31.783278 23292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.792220 23292 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:34677
I20260925 22:53:31.792294 23420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:34677 every 8 connection(s)
I20260925 22:53:31.793190 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:31.795894 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23292
I20260925 22:53:31.796067 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:31.796377 4974 external_mini_cluster.cc:1428] Setting key f12bd8d9f1c42f53321660e3eafc59c4
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:53:31.808214 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:31.794945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:49803 (local address 127.4.219.190:43821)
0925 22:53:31.795074 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:31.795078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:31.795862 (+ 784us) server_negotiation.cc:407] Connection header received
0925 22:53:31.797029 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:31.797032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:31.797093 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:31.797195 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:31.798583 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.799438 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.801299 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.801518 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.801599 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:31.804457 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:31.804463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:31.804468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:31.804504 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:31.806187 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.806700 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.806704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.806706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.806802 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:31.807176 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:31.807179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:31.807181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:31.807371 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:31.807452 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:31.807984 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 22:53:31.808087 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:53:31.809221 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:31.795274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43821 (local address 127.4.219.130:49803)
0925 22:53:31.795715 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:31.795758 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:31.796781 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:31.797384 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:31.797393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:31.797909 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:31.798430 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:31.798444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.799582 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.799588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:31.801180 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:31.801195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:31.801672 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:31.801680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:31.801913 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:31.802600 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:31.802624 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:31.804280 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:31.806311 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:31.806316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:31.806330 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:31.806597 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:31.806911 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:31.806915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:31.806920 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:31.807070 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:31.807463 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:31.807469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:31.807717 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 22:53:31.807930 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:53:31.808203 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
I20260925 22:53:31.810443 23421 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43821
I20260925 22:53:31.810691 23421 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:31.811180 23421 heartbeater.cc:511] Master 127.4.219.190:43821 requested a full tablet report, sending...
I20260925 22:53:31.812189 23096 ts_manager.cc:194] Registered new tserver with Master: 3e8a41d8c35c4a3db449aaf40aabd3bd (127.4.219.130:34677)
I20260925 22:53:31.812824 23096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:49803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:53:31.823568 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:31.934736 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:31.935075 23426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:31.935169 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:31.939468 23426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:31.939556 23426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:31.939678 23426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:31.945183 23426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:31.946270 23426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:31.947306 23426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:31.956938 23432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.956931 23431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:31.956926 23434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:31.957449 23426 server_base.cc:1034] running on GCE node
I20260925 22:53:31.957831 23426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:31.958448 23426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:31.959631 23426 hybrid_clock.cc:648] HybridClock initialized: now 1790376811959595 us; error 59 us; skew 500 ppm
Sep 25 22:53:31 dist-test-slave-kb1g krb5kdc[23051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:31.962345 23426 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:53:31.963224 23426 webserver.cc:460] Webserver started at http://127.4.219.131:36093/ using document root <none> and password file <none>
I20260925 22:53:31.963802 23426 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:31.963891 23426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:31.964146 23426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:31.966187 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "dbcc0797ea7b48b3976715b57540ad76"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db358bc6851ff39f691ef45bc15b1870"
server_key_iv: "7c59896c04adea3932bb93f715c48a46"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.966741 23426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "dbcc0797ea7b48b3976715b57540ad76"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db358bc6851ff39f691ef45bc15b1870"
server_key_iv: "7c59896c04adea3932bb93f715c48a46"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.970542 23426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 22:53:31.972954 23441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.974184 23426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:31.974316 23426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "dbcc0797ea7b48b3976715b57540ad76"
format_stamp: "Formatted at 2026-09-25 22:53:31 on dist-test-slave-kb1g"
server_key: "db358bc6851ff39f691ef45bc15b1870"
server_key_iv: "7c59896c04adea3932bb93f715c48a46"
server_key_version: "encryptionkey@0"
I20260925 22:53:31.974432 23426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:31.989669 23426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:31.992674 23426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:31.992877 23426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:31.993470 23426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:31.994285 23426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:31.994354 23426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:31.994427 23426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:31.994473 23426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.003360 23426 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:41227
I20260925 22:53:32.003376 23554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:41227 every 8 connection(s)
I20260925 22:53:32.004302 23426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:32.010031 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23426
I20260925 22:53:32.010166 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:32.010443 4974 external_mini_cluster.cc:1428] Setting key f11fa1ecaf35d9b54334de71eb71325a
Sep 25 22:53:32 dist-test-slave-kb1g krb5kdc[23051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:32.018350 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.006266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:45527 (local address 127.4.219.190:43821)
0925 22:53:32.006483 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.006486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.007286 (+ 800us) server_negotiation.cc:407] Connection header received
0925 22:53:32.008253 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.008256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.008325 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.008434 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:32.009679 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.010510 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.012007 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.012311 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.012411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.014883 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:32.014886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:32.014889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:32.014927 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:32.016214 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:32.016806 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:32.016811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:32.016813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:32.016908 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:32.017263 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:32.017266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:32.017268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:32.017454 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:32.017561 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.018055 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.018181 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 22:53:32.019163 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.006655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43821 (local address 127.4.219.131:45527)
0925 22:53:32.007127 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.007172 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.008035 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.008642 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.008649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.009023 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:32.009528 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.009542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.010665 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.010669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.011884 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.011895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.012486 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.012491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.012705 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.013244 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:32.013260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:32.014717 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:32.016335 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:32.016348 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:32.016366 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:32.016704 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:32.017009 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:32.017011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:32.017014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:32.017158 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:32.017558 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:32.017565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:32.017814 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.018001 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.018325 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":109,"threads_started":1}
I20260925 22:53:32.020252 23555 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43821
I20260925 22:53:32.020483 23555 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:32.020998 23555 heartbeater.cc:511] Master 127.4.219.190:43821 requested a full tablet report, sending...
I20260925 22:53:32.021968 23096 ts_manager.cc:194] Registered new tserver with Master: dbcc0797ea7b48b3976715b57540ad76 (127.4.219.131:41227)
I20260925 22:53:32.022512 23096 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:45527
I20260925 22:53:32.024622 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:32.034430 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.026577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.4.219.190:43821)
0925 22:53:32.026732 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.026737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.026925 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:53:32.027080 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.027084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.027129 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.027214 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:32.027899 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.028413 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.029257 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.029479 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.029590 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.031219 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:32.031224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:32.031231 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:32.031259 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:32.032467 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:32.033085 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:32.033089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:32.033091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:32.033175 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:32.033507 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:32.033511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:32.033513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:32.033721 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:32.033886 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.034155 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.034248 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:53:32.037251 23096 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:32.040580 23096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:32.055517 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.4.219.130:34677)
0925 22:53:32.050425 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.050439 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.050466 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:53:32.050548 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.050552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.050713 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.050839 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:32.051171 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052456 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.053979 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.054821 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.054935 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055106 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.055264 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.055358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":164,"threads_started":1}
I20260925 22:53:32.055537 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:34677 (local address 127.0.0.1:42996)
0925 22:53:32.049956 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.050009 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.050132 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.050854 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.050860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.050882 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:32.051038 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.051046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052763 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.052766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.053856 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.053865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.054965 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.054969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.055074 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055110 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.055222 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.055348 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":49,"thread_start_us":62,"threads_started":1}
I20260925 22:53:32.055860 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.4.219.131:41227)
0925 22:53:32.050149 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.050162 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.050627 (+ 465us) server_negotiation.cc:407] Connection header received
0925 22:53:32.050697 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.050700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.050892 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.051050 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:32.051281 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052431 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.054164 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.054942 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.055031 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055139 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.055616 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.055709 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":88,"threads_started":1}
I20260925 22:53:32.056870 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:41227 (local address 127.0.0.1:37720)
0925 22:53:32.050415 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.050429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.050523 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.051038 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.051041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.051059 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:32.051153 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.051158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052585 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.052589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.054045 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.054055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.055223 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.055227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.055314 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055338 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.055407 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.055475 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260925 22:53:32.057139 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.4.219.129:39845)
0925 22:53:32.051118 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.051133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.051154 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:53:32.051226 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.051231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.051387 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.051531 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:32.051697 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052619 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.054296 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.054853 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.054979 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055114 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.055783 (+ 669us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.055887 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":105,"threads_started":1}
I20260925 22:53:32.055537 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.049218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:39845 (local address 127.0.0.1:47488)
0925 22:53:32.050288 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.050302 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.050399 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.051509 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.051512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.051529 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:32.051609 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.051614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.052763 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.052766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.054182 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.054190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.055172 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.055178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.055266 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.055285 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.055357 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.055409 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":256,"thread_start_us":96,"threads_started":1}
I20260925 22:53:32.058861 23355 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 (DEFAULT_TABLE table=test-table [id=ef9a30b8ce334179852d9baec48efdbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:32.059185 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 (DEFAULT_TABLE table=test-table [id=ef9a30b8ce334179852d9baec48efdbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:32.059924 23355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6fdff94c9540c793b72038d3eeadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:32.060209 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6fdff94c9540c793b72038d3eeadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:32.062064 23221 tablet_service.cc:1467] Processing CreateTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 (DEFAULT_TABLE table=test-table [id=ef9a30b8ce334179852d9baec48efdbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:32.063066 23221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e6fdff94c9540c793b72038d3eeadd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:32.066246 23572 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Bootstrap starting.
I20260925 22:53:32.067920 23573 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Bootstrap starting.
I20260925 22:53:32.068297 23572 tablet_bootstrap.cc:654] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:32.069424 23572 log.cc:824] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:32.069864 23574 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: Bootstrap starting.
I20260925 22:53:32.070631 23573 tablet_bootstrap.cc:654] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:32.071411 23573 log.cc:824] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:32.071467 23572 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: No bootstrap required, opened a new log
I20260925 22:53:32.071643 23572 ts_tablet_manager.cc:1397] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:53:32.072141 23574 tablet_bootstrap.cc:654] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:32.073206 23573 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: No bootstrap required, opened a new log
I20260925 22:53:32.073380 23573 ts_tablet_manager.cc:1397] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:53:32.073396 23574 log.cc:824] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:32.075286 23574 tablet_bootstrap.cc:492] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: No bootstrap required, opened a new log
I20260925 22:53:32.075493 23574 ts_tablet_manager.cc:1397] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260925 22:53:32.075522 23572 raft_consensus.cc:348] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.075796 23572 raft_consensus.cc:374] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:32.075850 23572 raft_consensus.cc:729] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcc0797ea7b48b3976715b57540ad76, State: Initialized, Role: FOLLOWER
I20260925 22:53:32.076067 23573 raft_consensus.cc:348] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.076257 23573 raft_consensus.cc:374] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:32.076303 23573 raft_consensus.cc:729] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8a41d8c35c4a3db449aaf40aabd3bd, State: Initialized, Role: FOLLOWER
I20260925 22:53:32.076283 23572 consensus_queue.cc:260] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.077078 23555 heartbeater.cc:503] Master 127.4.219.190:43821 was elected leader, sending a full tablet report...
I20260925 22:53:32.077423 23572 ts_tablet_manager.cc:1428] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:53:32.076723 23573 consensus_queue.cc:260] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.078635 23421 heartbeater.cc:503] Master 127.4.219.190:43821 was elected leader, sending a full tablet report...
I20260925 22:53:32.079895 23573 ts_tablet_manager.cc:1428] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:53:32.080442 23574 raft_consensus.cc:348] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.080744 23574 raft_consensus.cc:374] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:32.080794 23574 raft_consensus.cc:729] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d92c2c9c1e649fbba08a9fc2410ebc6, State: Initialized, Role: FOLLOWER
I20260925 22:53:32.081243 23574 consensus_queue.cc:260] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.082160 23287 heartbeater.cc:503] Master 127.4.219.190:43821 was elected leader, sending a full tablet report...
I20260925 22:53:32.082278 23574 ts_tablet_manager.cc:1428] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:53:32.196002 23578 raft_consensus.cc:482] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:32.196235 23578 raft_consensus.cc:504] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.197513 23578 leader_election.cc:288] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8a41d8c35c4a3db449aaf40aabd3bd (127.4.219.130:34677), 1d92c2c9c1e649fbba08a9fc2410ebc6 (127.4.219.129:39845)
I20260925 22:53:32.201838 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.197825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:34677 (local address 127.4.219.131:38683)
0925 22:53:32.197933 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.197950 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.198068 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.198382 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.198385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.198404 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:32.198510 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.198514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.199501 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.199504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.200518 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.200580 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.201373 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.201376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.201500 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.201526 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.201614 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.201693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:53:32.201846 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.197930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:38683 (local address 127.4.219.130:34677)
0925 22:53:32.198082 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.198086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.198105 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:53:32.198212 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.198215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.198271 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.198356 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:32.198635 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.199386 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.200709 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.201259 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.201351 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.201392 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.201647 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.201723 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:53:32.202275 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.197825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:39845 (local address 127.4.219.131:50997)
0925 22:53:32.198650 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.198663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.198771 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.198999 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.199002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.199019 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:32.199131 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.199136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.200011 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.200014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.200935 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.200945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.201780 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.201783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.201875 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.201900 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.202074 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.202138 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":377,"thread_start_us":142,"threads_started":1}
I20260925 22:53:32.202294 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.198050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:50997 (local address 127.4.219.129:39845)
0925 22:53:32.198217 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.198222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.198766 (+ 544us) server_negotiation.cc:407] Connection header received
0925 22:53:32.198866 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.198868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.198911 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.198983 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:32.199234 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.199892 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.201089 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.201578 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.201683 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.201725 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.202107 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.202187 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:53:32.202508 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6fdff94c9540c793b72038d3eeadd8" candidate_uuid: "dbcc0797ea7b48b3976715b57540ad76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" is_pre_election: true
I20260925 22:53:32.202850 23375 raft_consensus.cc:2454] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbcc0797ea7b48b3976715b57540ad76 in term 0.
I20260925 22:53:32.202867 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6fdff94c9540c793b72038d3eeadd8" candidate_uuid: "dbcc0797ea7b48b3976715b57540ad76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" is_pre_election: true
I20260925 22:53:32.203150 23241 raft_consensus.cc:2454] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbcc0797ea7b48b3976715b57540ad76 in term 0.
I20260925 22:53:32.203385 23442 leader_election.cc:302] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a41d8c35c4a3db449aaf40aabd3bd, dbcc0797ea7b48b3976715b57540ad76; no voters:
I20260925 22:53:32.203678 23578 raft_consensus.cc:2781] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:32.203763 23578 raft_consensus.cc:482] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:32.203807 23578 raft_consensus.cc:3037] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:32.204828 23578 raft_consensus.cc:504] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.205192 23578 leader_election.cc:288] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8a41d8c35c4a3db449aaf40aabd3bd (127.4.219.130:34677), 1d92c2c9c1e649fbba08a9fc2410ebc6 (127.4.219.129:39845)
I20260925 22:53:32.205612 23375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6fdff94c9540c793b72038d3eeadd8" candidate_uuid: "dbcc0797ea7b48b3976715b57540ad76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd"
I20260925 22:53:32.205626 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e6fdff94c9540c793b72038d3eeadd8" candidate_uuid: "dbcc0797ea7b48b3976715b57540ad76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6"
I20260925 22:53:32.205741 23241 raft_consensus.cc:3037] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:32.205716 23375 raft_consensus.cc:3037] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:32.206729 23375 raft_consensus.cc:2454] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbcc0797ea7b48b3976715b57540ad76 in term 1.
I20260925 22:53:32.206736 23241 raft_consensus.cc:2454] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbcc0797ea7b48b3976715b57540ad76 in term 1.
I20260925 22:53:32.207041 23442 leader_election.cc:302] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8a41d8c35c4a3db449aaf40aabd3bd, dbcc0797ea7b48b3976715b57540ad76; no voters:
I20260925 22:53:32.207284 23578 raft_consensus.cc:2781] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:32.207520 23578 raft_consensus.cc:686] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 1 LEADER]: Becoming Leader. State: Replica: dbcc0797ea7b48b3976715b57540ad76, State: Running, Role: LEADER
I20260925 22:53:32.207862 23578 consensus_queue.cc:237] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } }
I20260925 22:53:32.211014 23096 catalog_manager.cc:5495] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbcc0797ea7b48b3976715b57540ad76 (127.4.219.131). New cstate: current_term: 1 leader_uuid: "dbcc0797ea7b48b3976715b57540ad76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbcc0797ea7b48b3976715b57540ad76" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 41227 } health_report { overall_health: HEALTHY } } }
I20260925 22:53:32.247718 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.244208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.4.219.131:41227)
0925 22:53:32.244338 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.244342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.244355 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:53:32.244516 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.244518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.244606 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.244698 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:32.244955 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.245460 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.246262 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.246477 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.246551 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.246926 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:32.247339 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:32.247460 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.247553 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.247589 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:53:32.254113 23375 raft_consensus.cc:1261] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 1 FOLLOWER]: Refusing update from remote peer dbcc0797ea7b48b3976715b57540ad76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:32.254243 23241 raft_consensus.cc:1261] T 5e6fdff94c9540c793b72038d3eeadd8 P 1d92c2c9c1e649fbba08a9fc2410ebc6 [term 1 FOLLOWER]: Refusing update from remote peer dbcc0797ea7b48b3976715b57540ad76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:32.254884 23582 consensus_queue.cc:1035] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8a41d8c35c4a3db449aaf40aabd3bd" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:32.255065 23578 consensus_queue.cc:1035] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d92c2c9c1e649fbba08a9fc2410ebc6" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:53:32.255358 23556 tablet.cc:2369] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:32.263259 23585 mvcc.cc:205] Tried to move back new op lower bound from 7333383422985072640 to 7333383422806196224. Current Snapshot: MvccSnapshot[applied={T|T < 7333383422985072640}]
I20260925 22:53:32.271936 23096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:32.272081 23096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:32.274546 23096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f430e48dfb4ef1b08ca9ad4e13c0f7: Sending DeleteTablet for 3 replicas of tablet 5e6fdff94c9540c793b72038d3eeadd8
I20260925 22:53:32.275315 23355 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:32 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:42996
I20260925 22:53:32.275367 23221 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:32 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:47488
I20260925 22:53:32.275789 23596 tablet_replica.cc:330] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: stopping tablet replica
I20260925 22:53:32.275828 23489 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e6fdff94c9540c793b72038d3eeadd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:32 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:37720
I20260925 22:53:32.276103 23596 raft_consensus.cc:2229] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:32.276450 23598 tablet_replica.cc:330] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: stopping tablet replica
I20260925 22:53:32.276463 23596 raft_consensus.cc:2258] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:32.276903 23598 raft_consensus.cc:2229] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:53:32.277303 23598 raft_consensus.cc:2258] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:32.277309 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23158
I20260925 22:53:32.278095 23596 ts_tablet_manager.cc:1905] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:32.280319 23598 ts_tablet_manager.cc:1905] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:32.280997 23596 ts_tablet_manager.cc:1918] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:32.281102 23596 log.cc:1196] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/5e6fdff94c9540c793b72038d3eeadd8
I20260925 22:53:32.281445 23596 ts_tablet_manager.cc:1939] T 5e6fdff94c9540c793b72038d3eeadd8 P 3e8a41d8c35c4a3db449aaf40aabd3bd: Deleting consensus metadata
I20260925 22:53:32.282756 23080 catalog_manager.cc:4841] TS 3e8a41d8c35c4a3db449aaf40aabd3bd (127.4.219.130:34677): tablet 5e6fdff94c9540c793b72038d3eeadd8 (table test-table [id=ef9a30b8ce334179852d9baec48efdbd]) successfully deleted
I20260925 22:53:32.283406 23598 ts_tablet_manager.cc:1918] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:32.283519 23598 log.cc:1196] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/5e6fdff94c9540c793b72038d3eeadd8
I20260925 22:53:32.283805 23598 ts_tablet_manager.cc:1939] T 5e6fdff94c9540c793b72038d3eeadd8 P dbcc0797ea7b48b3976715b57540ad76: Deleting consensus metadata
I20260925 22:53:32.285377 23081 catalog_manager.cc:4841] TS dbcc0797ea7b48b3976715b57540ad76 (127.4.219.131:41227): tablet 5e6fdff94c9540c793b72038d3eeadd8 (table test-table [id=ef9a30b8ce334179852d9baec48efdbd]) successfully deleted
W20260925 22:53:32.288897 23082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:39845 (error 108)
I20260925 22:53:32.289113 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23292
I20260925 22:53:32.290617 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.290089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:39845 (local address 127.0.0.1:47492)
0925 22:53:32.290288 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.290509 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:39845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 22:53:32.290791 23082 catalog_manager.cc:4584] TS 1d92c2c9c1e649fbba08a9fc2410ebc6 (127.4.219.129:39845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e6fdff94c9540c793b72038d3eeadd8: Network error: Client connection negotiation failed: client connection to 127.4.219.129:39845: connect: Connection refused (error 111)
I20260925 22:53:32.295410 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23426
I20260925 22:53:32.303381 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23064
2026-09-25T22:53:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2325 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:32Z Disabled control of system clock
I20260925 22:53:32.333954 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45745
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:32.446538 23603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:32.446811 23603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:32.446846 23603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:32.451015 23603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:32.451078 23603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:32.451094 23603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:32.451112 23603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:32.451150 23603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:32.456067 23603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:45745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:32.457211 23603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:32.458606 23603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:32.466852 23609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:32.466847 23611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:32.466857 23608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:32.467100 23603 server_base.cc:1034] running on GCE node
I20260925 22:53:32.467732 23603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:32.468822 23603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:32.470013 23603 hybrid_clock.cc:648] HybridClock initialized: now 1790376812470001 us; error 25 us; skew 500 ppm
I20260925 22:53:32.471843 23603 webserver.cc:460] Webserver started at http://127.4.219.190:43133/ using document root <none> and password file <none>
I20260925 22:53:32.472410 23603 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:32.472510 23603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:32.472793 23603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:32.474800 23603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "9a387f8ac796402a96257f5e299ac19a"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "48a2189128fa58089310b3131fc412a2"
server_key_iv: "fa3b29122ae41010ab194afb148a553f"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.475355 23603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "9a387f8ac796402a96257f5e299ac19a"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "48a2189128fa58089310b3131fc412a2"
server_key_iv: "fa3b29122ae41010ab194afb148a553f"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.479015 23603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260925 22:53:32.481398 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.482682 23603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:53:32.482817 23603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "9a387f8ac796402a96257f5e299ac19a"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "48a2189128fa58089310b3131fc412a2"
server_key_iv: "fa3b29122ae41010ab194afb148a553f"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.482928 23603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:32.492024 23603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:32.492699 23603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:32.492888 23603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:32.499481 23603 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:45745
I20260925 22:53:32.499511 23669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:45745 every 8 connection(s)
I20260925 22:53:32.500382 23603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:32.503427 23670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:32.508828 23670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: Bootstrap starting.
I20260925 22:53:32.509652 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23603
I20260925 22:53:32.509774 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:32.510067 4974 external_mini_cluster.cc:1428] Setting key 628832bb02d07222b93a993935ee3888
I20260925 22:53:32.511847 23670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:32.512717 23670 log.cc:824] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:32.514782 23670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: No bootstrap required, opened a new log
I20260925 22:53:32.519220 23670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } }
I20260925 22:53:32.519474 23670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:32.519536 23670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a387f8ac796402a96257f5e299ac19a, State: Initialized, Role: FOLLOWER
I20260925 22:53:32.519918 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.511330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.4.219.190:45745)
0925 22:53:32.512084 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.512100 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.512137 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:53:32.512915 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.512936 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.513344 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.513633 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:32.514001 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.515064 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.516270 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.516836 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.516967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.517844 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:32.517849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:32.517863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:32.518267 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:32.518328 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:32.518349 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:32.518424 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.518631 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.518971 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":133,"threads_started":1}
I20260925 22:53:32.519953 23670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } }
I20260925 22:53:32.520112 23670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:32.520174 23670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:32.520254 23670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:32.521255 23670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } }
I20260925 22:53:32.521621 23670 leader_election.cc:302] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a387f8ac796402a96257f5e299ac19a; no voters:
I20260925 22:53:32.521982 23670 leader_election.cc:288] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:32.522254 23675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:32.522668 23675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [term 1 LEADER]: Becoming Leader. State: Replica: 9a387f8ac796402a96257f5e299ac19a, State: Running, Role: LEADER
I20260925 22:53:32.523182 23675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } }
I20260925 22:53:32.523240 23670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:32.524968 23676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a387f8ac796402a96257f5e299ac19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } } }
I20260925 22:53:32.525046 23677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a387f8ac796402a96257f5e299ac19a. Latest consensus state: current_term: 1 leader_uuid: "9a387f8ac796402a96257f5e299ac19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a387f8ac796402a96257f5e299ac19a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 45745 } } }
I20260925 22:53:32.525110 23677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:32.525097 23676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:32.525553 23684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:32.528367 23684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:32.533423 23684 catalog_manager.cc:1347] Generated new cluster ID: cdb9de2ff9ef45faab72b89b061050f9
I20260925 22:53:32.533492 23684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:32.548579 23684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:32.549643 23684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:32.554780 23684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: Generated new TSK 0
I20260925 22:53:32.555490 23684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:32.561738 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--builtin_ntp_servers=127.4.219.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:32.674643 23694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:32.674912 23694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:32.674952 23694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:32.679153 23694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:32.679208 23694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:32.679298 23694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:32.684078 23694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:32.685213 23694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:32.686627 23694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:32.696449 23699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:32.696444 23700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:32.696693 23694 server_base.cc:1034] running on GCE node
W20260925 22:53:32.696444 23702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:32.697145 23694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:32.697866 23694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:32.699044 23694 hybrid_clock.cc:648] HybridClock initialized: now 1790376812699026 us; error 27 us; skew 500 ppm
I20260925 22:53:32.700927 23694 webserver.cc:460] Webserver started at http://127.4.219.129:36003/ using document root <none> and password file <none>
I20260925 22:53:32.701540 23694 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:32.701608 23694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:32.701779 23694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:32.703794 23694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "c1f6fc564225481381736e608dbc811b"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "99ce41e62fd89eb11a9828d3cf3341d9"
server_key_iv: "d76d4d706c07c8e09f41b41652cd6d75"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.704305 23694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "c1f6fc564225481381736e608dbc811b"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "99ce41e62fd89eb11a9828d3cf3341d9"
server_key_iv: "d76d4d706c07c8e09f41b41652cd6d75"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.708026 23694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:53:32.710512 23708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.711766 23694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:32.711905 23694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "c1f6fc564225481381736e608dbc811b"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "99ce41e62fd89eb11a9828d3cf3341d9"
server_key_iv: "d76d4d706c07c8e09f41b41652cd6d75"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.712018 23694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:32.742210 23694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:32.743026 23694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:32.743250 23694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:32.743927 23694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:32.745041 23694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:32.745116 23694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.745194 23694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:32.745250 23694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.754878 23694 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:37567
I20260925 22:53:32.754901 23821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:37567 every 8 connection(s)
I20260925 22:53:32.755896 23694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:32.758359 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23694
I20260925 22:53:32.758535 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:32.758838 4974 external_mini_cluster.cc:1428] Setting key b3e46bcc05f2b49b30b202f9e5196bf3
I20260925 22:53:32.761179 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--builtin_ntp_servers=127.4.219.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:32.766484 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.757718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:56363 (local address 127.4.219.190:45745)
0925 22:53:32.757871 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.757875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.759055 (+ 1180us) server_negotiation.cc:407] Connection header received
0925 22:53:32.760975 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.760981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.761054 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.761157 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:32.762152 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.762837 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.764306 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.764752 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.764877 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.765984 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:32.765988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:32.765990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:32.766003 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:32.766029 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:32.766037 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:32.766114 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.766284 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.766345 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:53:32.767457 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.758159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45745 (local address 127.4.219.129:56363)
0925 22:53:32.758887 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.758940 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.760742 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.761382 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.761394 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.761497 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:32.762002 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.762015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.762987 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.762994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.764158 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.764168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.764925 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.764930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.765107 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.765722 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:32.765723 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:32.765762 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:32.765824 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:32.766131 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:32.766147 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:32.766152 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.766269 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.766555 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260925 22:53:32.768725 23822 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45745
I20260925 22:53:32.768949 23822 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:32.769455 23822 heartbeater.cc:511] Master 127.4.219.190:45745 requested a full tablet report, sending...
I20260925 22:53:32.770892 23634 ts_manager.cc:194] Registered new tserver with Master: c1f6fc564225481381736e608dbc811b (127.4.219.129:37567)
I20260925 22:53:32.772027 23634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.129:56363
W20260925 22:53:32.891568 23825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:32.891898 23825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:32.891938 23825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:32.896335 23825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:32.896409 23825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:32.896467 23825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:32.901465 23825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:32.902612 23825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:32.904054 23825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:32.913750 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:32.913818 23833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:32.913869 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:32.914002 23825 server_base.cc:1034] running on GCE node
I20260925 22:53:32.914638 23825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:32.915350 23825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:32.916585 23825 hybrid_clock.cc:648] HybridClock initialized: now 1790376812916540 us; error 56 us; skew 500 ppm
I20260925 22:53:32.924108 23825 webserver.cc:460] Webserver started at http://127.4.219.130:46875/ using document root <none> and password file <none>
I20260925 22:53:32.924860 23825 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:32.924927 23825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:32.925163 23825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:32.927471 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "a4492a7a87dc4f9fa2641d936f51fa5c"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "24fa94cc214c6c21e2a227cd1a8ead0d"
server_key_iv: "444546625f73ed5b7c5187ddccc1ba7e"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.928069 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "a4492a7a87dc4f9fa2641d936f51fa5c"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "24fa94cc214c6c21e2a227cd1a8ead0d"
server_key_iv: "444546625f73ed5b7c5187ddccc1ba7e"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.932035 23825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:53:32.934705 23839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.936173 23825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:53:32.936347 23825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "a4492a7a87dc4f9fa2641d936f51fa5c"
format_stamp: "Formatted at 2026-09-25 22:53:32 on dist-test-slave-kb1g"
server_key: "24fa94cc214c6c21e2a227cd1a8ead0d"
server_key_iv: "444546625f73ed5b7c5187ddccc1ba7e"
server_key_version: "encryptionkey@0"
I20260925 22:53:32.936492 23825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:32.951541 23825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:32.952283 23825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:32.952498 23825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:32.953152 23825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:32.954089 23825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:32.954159 23825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.954228 23825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:32.954272 23825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:32.963032 23825 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44317
I20260925 22:53:32.963068 23952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44317 every 8 connection(s)
I20260925 22:53:32.963967 23825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:32.968236 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23825
I20260925 22:53:32.968376 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:32.968743 4974 external_mini_cluster.cc:1428] Setting key 0ed0bee60b66460bc8880de730a48727
I20260925 22:53:32.970894 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--builtin_ntp_servers=127.4.219.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:32.974360 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.965823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:33373 (local address 127.4.219.190:45745)
0925 22:53:32.965960 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:32.965964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:32.966963 (+ 999us) server_negotiation.cc:407] Connection header received
0925 22:53:32.968605 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:32.968609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:32.968660 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:32.968736 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:32.969837 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.970464 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.972243 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.972670 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.972782 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.973863 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:32.973873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:32.973876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:32.973887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:32.973904 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:32.973912 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:32.973995 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:32.974159 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 22:53:32.974220 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:53:32.975297 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:32.966188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45745 (local address 127.4.219.130:33373)
0925 22:53:32.966801 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:32.966841 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:32.968278 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:32.968960 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:32.968969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:32.969075 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:32.969698 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.969718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.970608 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.970613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.972097 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:32.972111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:32.972815 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:32.972818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:32.972963 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:32.973607 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:32.973609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:32.973649 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:32.973712 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:32.973997 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:32.974008 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:32.974017 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:53:32.974148 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 22:53:32.974431 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
I20260925 22:53:32.976428 23953 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45745
I20260925 22:53:32.976711 23953 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:32.977247 23953 heartbeater.cc:511] Master 127.4.219.190:45745 requested a full tablet report, sending...
I20260925 22:53:32.978331 23634 ts_manager.cc:194] Registered new tserver with Master: a4492a7a87dc4f9fa2641d936f51fa5c (127.4.219.130:44317)
I20260925 22:53:32.979246 23634 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.130:33373
W20260925 22:53:33.082304 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:33.082552 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:33.082588 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:33.086649 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:33.086697 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:33.086750 23956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:33.091503 23956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:33.092612 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:33.094027 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:33.103369 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:33.103417 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:33.103319 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:33.103502 23956 server_base.cc:1034] running on GCE node
I20260925 22:53:33.104039 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:33.104795 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:33.105976 23956 hybrid_clock.cc:648] HybridClock initialized: now 1790376813105942 us; error 50 us; skew 500 ppm
I20260925 22:53:33.107898 23956 webserver.cc:460] Webserver started at http://127.4.219.131:39947/ using document root <none> and password file <none>
I20260925 22:53:33.108489 23956 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:33.108613 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:33.108850 23956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:33.110906 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7"
format_stamp: "Formatted at 2026-09-25 22:53:33 on dist-test-slave-kb1g"
server_key: "3874344a6660bc677f8e2395a28add7b"
server_key_iv: "6896c99df0be5563b18e9216ceb1167d"
server_key_version: "encryptionkey@0"
I20260925 22:53:33.111487 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7"
format_stamp: "Formatted at 2026-09-25 22:53:33 on dist-test-slave-kb1g"
server_key: "3874344a6660bc677f8e2395a28add7b"
server_key_iv: "6896c99df0be5563b18e9216ceb1167d"
server_key_version: "encryptionkey@0"
I20260925 22:53:33.115186 23956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260925 22:53:33.117750 23970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:33.119076 23956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:33.119170 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7"
format_stamp: "Formatted at 2026-09-25 22:53:33 on dist-test-slave-kb1g"
server_key: "3874344a6660bc677f8e2395a28add7b"
server_key_iv: "6896c99df0be5563b18e9216ceb1167d"
server_key_version: "encryptionkey@0"
I20260925 22:53:33.119259 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:33.138019 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:33.138628 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:33.138787 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:33.139283 23956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:33.140231 23956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:33.140277 23956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:33.140322 23956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:33.140343 23956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:33.149024 23956 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:35481
I20260925 22:53:33.149063 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:35481 every 8 connection(s)
I20260925 22:53:33.149891 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:33.157689 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 23956
I20260925 22:53:33.157825 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:33.158169 4974 external_mini_cluster.cc:1428] Setting key 125e1e604c4a964d55a409bf88a0f751
I20260925 22:53:33.158593 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.151599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44225 (local address 127.4.219.190:45745)
0925 22:53:33.151737 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.151740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.152758 (+ 1018us) server_negotiation.cc:407] Connection header received
0925 22:53:33.154061 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.154063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.154108 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.154179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:33.154978 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.155477 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.156595 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.156812 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.156889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.158035 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:33.158040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:33.158043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:33.158057 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:33.158078 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:33.158088 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:33.158167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.158406 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.158465 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:53:33.159619 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.151960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:45745 (local address 127.4.219.131:44225)
0925 22:53:33.152619 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.152657 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.153886 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.154312 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.154318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.154397 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:33.154855 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.154867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.155580 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.155582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.156445 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.156453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.156939 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.156942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.157080 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.157716 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:33.157718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:33.157755 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:33.157868 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:33.158203 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:33.158211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:33.158216 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.158382 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.158659 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":106,"threads_started":1}
I20260925 22:53:33.160701 24084 heartbeater.cc:347] Connected to a master server at 127.4.219.190:45745
I20260925 22:53:33.160924 24084 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:33.161403 24084 heartbeater.cc:511] Master 127.4.219.190:45745 requested a full tablet report, sending...
I20260925 22:53:33.162295 23633 ts_manager.cc:194] Registered new tserver with Master: 1a0bb65a1d4b4d71a2d7fe6df27203b7 (127.4.219.131:35481)
I20260925 22:53:33.162820 23633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.131:44225
I20260925 22:53:33.172164 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:33.177615 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.174185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.4.219.190:45745)
0925 22:53:33.174305 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.174310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.174451 (+ 141us) server_negotiation.cc:407] Connection header received
0925 22:53:33.174549 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.174551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.174588 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.174646 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:33.174885 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.175389 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.176191 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.176391 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.176463 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.177257 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:33.177260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:33.177262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:33.177272 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:33.177289 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:33.177297 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:33.177388 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.177483 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.177513 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:53:33.180382 23633 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:33.183420 23633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:33.197702 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.191513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:35481 (local address 127.0.0.1:46012)
0925 22:53:33.192218 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.192265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.192386 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.193449 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.193452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.193469 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:33.193577 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.193584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.195079 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.195081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.196105 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.196114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197170 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.197290 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.197327 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.197433 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.197560 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":217,"thread_start_us":89,"threads_started":1}
I20260925 22:53:33.197999 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.191627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.4.219.130:44317)
0925 22:53:33.192705 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.192718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.192742 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:53:33.192819 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.192826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.193038 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.193197 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:33.193505 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.194619 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.196458 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197208 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197302 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.197492 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.197729 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.197841 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":129,"threads_started":1}
I20260925 22:53:33.198000 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.191508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44317 (local address 127.0.0.1:46948)
0925 22:53:33.192226 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.192300 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.192385 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.193194 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.193200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.193223 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:33.193357 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.193368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.194840 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.194847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.196318 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.196328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197427 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.197565 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.197594 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.197694 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.197864 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":59,"thread_start_us":62,"threads_started":1}
I20260925 22:53:33.198349 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.192208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:37567 (local address 127.0.0.1:49414)
0925 22:53:33.192648 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.192660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.192738 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.193617 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.193620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.193636 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:33.193709 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.193714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.195253 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.195256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.196341 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.196347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197422 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.198016 (+ 588us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.198065 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.198147 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.198226 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":473,"thread_start_us":86,"threads_started":1}
I20260925 22:53:33.198730 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.192736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.4.219.129:37567)
0925 22:53:33.193126 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.193139 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.193163 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:53:33.193232 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.193236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.193435 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.193578 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:33.193821 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.194914 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.196464 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197048 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197172 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.197349 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.198534 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.198617 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":105,"threads_started":1}
I20260925 22:53:33.199018 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.191735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.4.219.131:35481)
0925 22:53:33.193010 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.193025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.193045 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:53:33.193103 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.193107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.193307 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.193459 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:33.193706 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.194911 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.196228 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.197034 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.197148 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.197350 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.197546 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.197641 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":142,"threads_started":1}
I20260925 22:53:33.201294 24018 tablet_service.cc:1467] Processing CreateTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc (DEFAULT_TABLE table=test-table [id=9e6709de443c4cdd9094b0d33ed66615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:33.201128 23887 tablet_service.cc:1467] Processing CreateTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc (DEFAULT_TABLE table=test-table [id=9e6709de443c4cdd9094b0d33ed66615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:33.201542 23756 tablet_service.cc:1467] Processing CreateTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc (DEFAULT_TABLE table=test-table [id=9e6709de443c4cdd9094b0d33ed66615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:33.202214 24018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b372dae1f94bc5898711fdc6a6e6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:33.202247 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b372dae1f94bc5898711fdc6a6e6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:33.202383 23756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54b372dae1f94bc5898711fdc6a6e6cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:33.208436 24101 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Bootstrap starting.
I20260925 22:53:33.209515 24103 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Bootstrap starting.
I20260925 22:53:33.209858 24102 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Bootstrap starting.
I20260925 22:53:33.210824 24101 tablet_bootstrap.cc:654] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:33.211535 24102 tablet_bootstrap.cc:654] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:33.211549 24103 tablet_bootstrap.cc:654] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:33.211650 24101 log.cc:824] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:33.212281 24103 log.cc:824] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:33.212340 24102 log.cc:824] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:33.213677 24101 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: No bootstrap required, opened a new log
I20260925 22:53:33.213858 24101 ts_tablet_manager.cc:1397] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:53:33.214174 24103 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: No bootstrap required, opened a new log
I20260925 22:53:33.214363 24103 ts_tablet_manager.cc:1397] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:53:33.214378 24102 tablet_bootstrap.cc:492] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: No bootstrap required, opened a new log
I20260925 22:53:33.214538 24102 ts_tablet_manager.cc:1397] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:53:33.217773 24103 raft_consensus.cc:348] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.218017 24103 raft_consensus.cc:374] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:33.218070 24103 raft_consensus.cc:729] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4492a7a87dc4f9fa2641d936f51fa5c, State: Initialized, Role: FOLLOWER
I20260925 22:53:33.218156 24101 raft_consensus.cc:348] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.218400 24101 raft_consensus.cc:374] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:33.218465 24101 raft_consensus.cc:729] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0bb65a1d4b4d71a2d7fe6df27203b7, State: Initialized, Role: FOLLOWER
I20260925 22:53:33.218559 24103 consensus_queue.cc:260] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.218567 24102 raft_consensus.cc:348] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.218806 24102 raft_consensus.cc:374] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:33.218868 24102 raft_consensus.cc:729] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f6fc564225481381736e608dbc811b, State: Initialized, Role: FOLLOWER
I20260925 22:53:33.218886 24101 consensus_queue.cc:260] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.219319 24102 consensus_queue.cc:260] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.219617 24103 ts_tablet_manager.cc:1428] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:53:33.219707 24084 heartbeater.cc:503] Master 127.4.219.190:45745 was elected leader, sending a full tablet report...
I20260925 22:53:33.219853 24101 ts_tablet_manager.cc:1428] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:53:33.219962 23953 heartbeater.cc:503] Master 127.4.219.190:45745 was elected leader, sending a full tablet report...
I20260925 22:53:33.220934 23822 heartbeater.cc:503] Master 127.4.219.190:45745 was elected leader, sending a full tablet report...
I20260925 22:53:33.222007 24102 ts_tablet_manager.cc:1428] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260925 22:53:33.257083 23823 tablet.cc:2369] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:33.261935 24107 raft_consensus.cc:482] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:33.262162 24107 raft_consensus.cc:504] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.263264 24107 leader_election.cc:288] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a0bb65a1d4b4d71a2d7fe6df27203b7 (127.4.219.131:35481), c1f6fc564225481381736e608dbc811b (127.4.219.129:37567)
I20260925 22:53:33.266969 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.263556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:35481 (local address 127.4.219.130:50935)
0925 22:53:33.263701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.263720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.263826 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.264223 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.264226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.264252 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:33.264346 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.264351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.265148 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.265154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.265927 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.265935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.266558 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.266657 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.266684 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.266771 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.266839 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:53:33.266968 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.263654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:50935 (local address 127.4.219.131:35481)
0925 22:53:33.263798 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.263803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.263818 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:53:33.263955 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.263961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.264032 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.264110 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:33.264461 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.265038 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266049 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.266441 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266567 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.266616 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.266786 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.266849 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:53:33.267380 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.264008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:56613 (local address 127.4.219.129:37567)
0925 22:53:33.264234 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.264240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.264330 (+ 90us) server_negotiation.cc:407] Connection header received
0925 22:53:33.264493 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.264497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.264609 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.264699 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:33.264971 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.265447 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266433 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.266824 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266949 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.266992 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.267189 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.267253 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 22:53:33.267661 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b372dae1f94bc5898711fdc6a6e6cc" candidate_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" is_pre_election: true
I20260925 22:53:33.267733 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.263883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:37567 (local address 127.4.219.130:56613)
0925 22:53:33.264230 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.264244 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.264372 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:33.264710 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:33.264713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:33.264728 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:33.264853 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.264859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.265559 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.265565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.266314 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:33.266322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.266947 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.266951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:33.267050 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.267074 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:53:33.267513 (+ 439us) client_negotiation.cc:240] Negotiation successful
0925 22:53:33.267609 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":139,"threads_started":1}
I20260925 22:53:33.267972 24038 raft_consensus.cc:2454] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4492a7a87dc4f9fa2641d936f51fa5c in term 0.
I20260925 22:53:33.268306 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b372dae1f94bc5898711fdc6a6e6cc" candidate_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f6fc564225481381736e608dbc811b" is_pre_election: true
I20260925 22:53:33.268525 23841 leader_election.cc:302] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a0bb65a1d4b4d71a2d7fe6df27203b7, a4492a7a87dc4f9fa2641d936f51fa5c; no voters:
I20260925 22:53:33.268621 23776 raft_consensus.cc:2454] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4492a7a87dc4f9fa2641d936f51fa5c in term 0.
I20260925 22:53:33.268851 24107 raft_consensus.cc:2781] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:33.268946 24107 raft_consensus.cc:482] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:33.268999 24107 raft_consensus.cc:3037] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:33.269989 24107 raft_consensus.cc:504] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.270320 24107 leader_election.cc:288] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [CANDIDATE]: Term 1 election: Requested vote from peers 1a0bb65a1d4b4d71a2d7fe6df27203b7 (127.4.219.131:35481), c1f6fc564225481381736e608dbc811b (127.4.219.129:37567)
I20260925 22:53:33.270646 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b372dae1f94bc5898711fdc6a6e6cc" candidate_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7"
I20260925 22:53:33.270735 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54b372dae1f94bc5898711fdc6a6e6cc" candidate_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f6fc564225481381736e608dbc811b"
I20260925 22:53:33.270809 24038 raft_consensus.cc:3037] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:33.270895 23776 raft_consensus.cc:3037] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:33.271839 24038 raft_consensus.cc:2454] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4492a7a87dc4f9fa2641d936f51fa5c in term 1.
I20260925 22:53:33.271956 23776 raft_consensus.cc:2454] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4492a7a87dc4f9fa2641d936f51fa5c in term 1.
I20260925 22:53:33.272143 23841 leader_election.cc:302] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a0bb65a1d4b4d71a2d7fe6df27203b7, a4492a7a87dc4f9fa2641d936f51fa5c; no voters:
I20260925 22:53:33.272316 24107 raft_consensus.cc:2781] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:33.272605 24107 raft_consensus.cc:686] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 1 LEADER]: Becoming Leader. State: Replica: a4492a7a87dc4f9fa2641d936f51fa5c, State: Running, Role: LEADER
I20260925 22:53:33.272957 24107 consensus_queue.cc:237] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } }
I20260925 22:53:33.276278 23632 catalog_manager.cc:5495] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to a4492a7a87dc4f9fa2641d936f51fa5c (127.4.219.130). New cstate: current_term: 1 leader_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4492a7a87dc4f9fa2641d936f51fa5c" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 } health_report { overall_health: UNKNOWN } } }
I20260925 22:53:33.295826 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:33.292240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.4.219.130:44317)
0925 22:53:33.292374 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:33.292379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:33.292395 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:53:33.292549 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:33.292552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:33.292611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:33.292700 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:33.292908 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.293397 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.294288 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:33.294530 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:33.294644 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:33.294954 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:33.295358 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:33.295529 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:33.295615 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:53:33.295651 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:53:33.301872 24038 raft_consensus.cc:1261] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 1 FOLLOWER]: Refusing update from remote peer a4492a7a87dc4f9fa2641d936f51fa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:33.301890 23776 raft_consensus.cc:1261] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b [term 1 FOLLOWER]: Refusing update from remote peer a4492a7a87dc4f9fa2641d936f51fa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:33.302556 24107 consensus_queue.cc:1035] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a0bb65a1d4b4d71a2d7fe6df27203b7" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:33.302804 24111 consensus_queue.cc:1035] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f6fc564225481381736e608dbc811b" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:33.312724 24117 mvcc.cc:205] Tried to move back new op lower bound from 7333383427276996608 to 7333383427168411648. Current Snapshot: MvccSnapshot[applied={T|T < 7333383427276996608}]
I20260925 22:53:33.315511 24115 mvcc.cc:205] Tried to move back new op lower bound from 7333383427276996608 to 7333383427168411648. Current Snapshot: MvccSnapshot[applied={T|T < 7333383427276996608}]
I20260925 22:53:33.319940 23632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:33.320081 23632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:33.322535 23632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a387f8ac796402a96257f5e299ac19a: Sending DeleteTablet for 3 replicas of tablet 54b372dae1f94bc5898711fdc6a6e6cc
I20260925 22:53:33.323307 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:33 UTC) from {username='slave'} at 127.0.0.1:46948
I20260925 22:53:33.323453 23756 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:33 UTC) from {username='slave'} at 127.0.0.1:49414
I20260925 22:53:33.323601 24018 tablet_service.cc:1514] Processing DeleteTablet for tablet 54b372dae1f94bc5898711fdc6a6e6cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:33 UTC) from {username='slave'} at 127.0.0.1:46012
I20260925 22:53:33.323930 24124 tablet_replica.cc:330] T 54b372dae1f94bc5898711fdc6a6e6cc P c1f6fc564225481381736e608dbc811b: stopping tablet replica
I20260925 22:53:33.324043 24125 tablet_replica.cc:330] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: stopping tablet replica
I20260925 22:53:33.324407 24123 tablet_replica.cc:330] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: stopping tablet replica
I20260925 22:53:33.324390 24125 raft_consensus.cc:2229] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:33.324708 24123 raft_consensus.cc:2229] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:53:33.324793 24125 raft_consensus.cc:2258] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:33.325026 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23694
I20260925 22:53:33.326771 24123 raft_consensus.cc:2258] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:33.327267 24125 ts_tablet_manager.cc:1905] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:33.328007 24123 ts_tablet_manager.cc:1905] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:33.330166 24125 ts_tablet_manager.cc:1918] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:33.330271 24125 log.cc:1196] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/54b372dae1f94bc5898711fdc6a6e6cc
I20260925 22:53:33.330602 24125 ts_tablet_manager.cc:1939] T 54b372dae1f94bc5898711fdc6a6e6cc P 1a0bb65a1d4b4d71a2d7fe6df27203b7: Deleting consensus metadata
I20260925 22:53:33.331220 24123 ts_tablet_manager.cc:1918] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:33.331318 24123 log.cc:1196] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/54b372dae1f94bc5898711fdc6a6e6cc
I20260925 22:53:33.331633 24123 ts_tablet_manager.cc:1939] T 54b372dae1f94bc5898711fdc6a6e6cc P a4492a7a87dc4f9fa2641d936f51fa5c: Deleting consensus metadata
I20260925 22:53:33.331802 23619 catalog_manager.cc:4841] TS 1a0bb65a1d4b4d71a2d7fe6df27203b7 (127.4.219.131:35481): tablet 54b372dae1f94bc5898711fdc6a6e6cc (table test-table [id=9e6709de443c4cdd9094b0d33ed66615]) successfully deleted
W20260925 22:53:33.332937 23619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:37567 (error 108)
I20260925 22:53:33.333546 23620 catalog_manager.cc:4841] TS a4492a7a87dc4f9fa2641d936f51fa5c (127.4.219.130:44317): tablet 54b372dae1f94bc5898711fdc6a6e6cc (table test-table [id=9e6709de443c4cdd9094b0d33ed66615]) successfully deleted
W20260925 22:53:33.334000 24099 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:53:33.333708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:37567 (local address 127.0.0.1:49418)
0925 22:53:33.333818 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:33.333833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:33.333905 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:37567: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:53:33.334146 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23825
W20260925 22:53:33.334162 23619 catalog_manager.cc:4584] TS c1f6fc564225481381736e608dbc811b (127.4.219.129:37567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54b372dae1f94bc5898711fdc6a6e6cc: Network error: Client connection negotiation failed: client connection to 127.4.219.129:37567: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 22:53:33.340061 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23956
I20260925 22:53:33.345443 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 23603
2026-09-25T22:53:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1040 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:33 dist-test-slave-kb1g krb5kdc[24129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:33 dist-test-slave-kb1g krb5kdc[24129](info): set up 2 sockets
Sep 25 22:53:33 dist-test-slave-kb1g krb5kdc[24129](info): commencing operation
krb5kdc: starting...
W20260925 22:53:34.464520 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:34 dist-test-slave-kb1g krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:34.581348 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:43215
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:34.689263 24142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:34.689538 24142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:34.689610 24142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:34.693506 24142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:34.693591 24142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:34.693627 24142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:34.693688 24142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:34.693724 24142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:34.699020 24142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:43215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:34.700088 24142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:34.701131 24142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:34.709743 24147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:34.709905 24142 server_base.cc:1034] running on GCE node
W20260925 22:53:34.709996 24150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:34.710075 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:34.710590 24142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:34.711665 24142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:34.712883 24142 hybrid_clock.cc:648] HybridClock initialized: now 1790376814712862 us; error 30 us; skew 500 ppm
Sep 25 22:53:34 dist-test-slave-kb1g krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:34.715603 24142 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:34.716485 24142 webserver.cc:460] Webserver started at http://127.4.219.190:36355/ using document root <none> and password file <none>
I20260925 22:53:34.717044 24142 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:34.717104 24142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:34.717260 24142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:34.719231 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "fde6462d867e4050bfed6c3eff7fc3c1"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "c30808d3b3260568a141d9ef4b30ae64"
server_key_iv: "e40b8fcec16a0bfe7544a8a3717fab58"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.719731 24142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "fde6462d867e4050bfed6c3eff7fc3c1"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "c30808d3b3260568a141d9ef4b30ae64"
server_key_iv: "e40b8fcec16a0bfe7544a8a3717fab58"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.723315 24142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:53:34.725697 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:34.726964 24142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:34.727061 24142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "fde6462d867e4050bfed6c3eff7fc3c1"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "c30808d3b3260568a141d9ef4b30ae64"
server_key_iv: "e40b8fcec16a0bfe7544a8a3717fab58"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.727145 24142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:34.742494 24142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:34.746064 24142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:34.746232 24142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:34.753655 24142 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:43215
I20260925 22:53:34.753667 24209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:43215 every 8 connection(s)
I20260925 22:53:34.754702 24142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:34.757025 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24142
I20260925 22:53:34.757174 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:34.757469 4974 external_mini_cluster.cc:1428] Setting key e92222f9990c2f428b6bf3c5611a844e
I20260925 22:53:34.757630 24210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:34.764765 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: Bootstrap starting.
Sep 25 22:53:34 dist-test-slave-kb1g krb5kdc[24129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:34.767091 24210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:34.767970 24210 log.cc:824] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:34.770291 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: No bootstrap required, opened a new log
I20260925 22:53:34.772698 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:34.758877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.4.219.190:43215)
0925 22:53:34.759294 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:34.759312 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:34.759352 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:53:34.760072 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:34.760094 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:34.760507 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:34.760893 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:34.761634 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:34.763093 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:34.764093 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:34.764773 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:34.764976 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:34.767497 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:34.767503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:34.767519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:34.767549 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:34.769623 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:34.770040 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:34.770046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:34.770051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:34.770208 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:34.770509 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:34.770513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:34.770514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:34.771014 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:34.771188 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:34.771396 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:53:34.771682 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260925 22:53:34.774606 24210 raft_consensus.cc:348] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } }
I20260925 22:53:34.774896 24210 raft_consensus.cc:374] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:34.774971 24210 raft_consensus.cc:729] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde6462d867e4050bfed6c3eff7fc3c1, State: Initialized, Role: FOLLOWER
I20260925 22:53:34.775411 24210 consensus_queue.cc:260] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } }
I20260925 22:53:34.775557 24210 raft_consensus.cc:388] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:34.775624 24210 raft_consensus.cc:482] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:34.775719 24210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:34.776731 24210 raft_consensus.cc:504] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } }
I20260925 22:53:34.777051 24210 leader_election.cc:302] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fde6462d867e4050bfed6c3eff7fc3c1; no voters:
I20260925 22:53:34.777371 24210 leader_election.cc:288] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:34.777485 24215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:34.777722 24215 raft_consensus.cc:686] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [term 1 LEADER]: Becoming Leader. State: Replica: fde6462d867e4050bfed6c3eff7fc3c1, State: Running, Role: LEADER
I20260925 22:53:34.778036 24215 consensus_queue.cc:237] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } }
I20260925 22:53:34.778581 24210 sys_catalog.cc:564] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:34.779157 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde6462d867e4050bfed6c3eff7fc3c1. Latest consensus state: current_term: 1 leader_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } } }
I20260925 22:53:34.779277 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:34.779699 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde6462d867e4050bfed6c3eff7fc3c1" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 43215 } } }
I20260925 22:53:34.779793 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1 [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:34.783402 24230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:34.783499 24230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:34.783576 24221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:34.784273 24221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:34.789165 24221 catalog_manager.cc:1347] Generated new cluster ID: 701381293d944c73ba2147cdab417fa2
I20260925 22:53:34.789225 24221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:34.805054 24221 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:34.806146 24221 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:34.811323 24221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: Generated new TSK 0
I20260925 22:53:34.812072 24221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.219.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
Entry for principal kudu/127.4.219.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab.
I20260925 22:53:34.843673 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:34.957083 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:34.957350 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:34.957394 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:34.961671 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:34.961735 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:34.961853 24236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:34.967407 24236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.129.keytab
--principal=kudu/127.4.219.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:34.968518 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:34.969689 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:34.979339 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:34.979337 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:34.979324 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:34.979580 24236 server_base.cc:1034] running on GCE node
I20260925 22:53:34.979995 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:34.980774 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:34.981988 24236 hybrid_clock.cc:648] HybridClock initialized: now 1790376814981971 us; error 28 us; skew 500 ppm
Sep 25 22:53:34 dist-test-slave-kb1g krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:34.984830 24236 init.cc:377] Logged in from keytab as kudu/127.4.219.129@KRBTEST.COM (short username kudu)
I20260925 22:53:34.985682 24236 webserver.cc:460] Webserver started at http://127.4.219.129:43865/ using document root <none> and password file <none>
I20260925 22:53:34.986198 24236 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:34.986255 24236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:34.986418 24236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:34.988377 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "864203a208de4990a65c026e0cf32ed8"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "fc5a2b70fdd3e26ff5ba03bc23cac26e"
server_key_iv: "c2ddbf7efdb6a6f1685c249db942ef42"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.988977 24236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "864203a208de4990a65c026e0cf32ed8"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "fc5a2b70fdd3e26ff5ba03bc23cac26e"
server_key_iv: "c2ddbf7efdb6a6f1685c249db942ef42"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.992744 24236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:53:34.995100 24251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:34.996299 24236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:34.996389 24236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "864203a208de4990a65c026e0cf32ed8"
format_stamp: "Formatted at 2026-09-25 22:53:34 on dist-test-slave-kb1g"
server_key: "fc5a2b70fdd3e26ff5ba03bc23cac26e"
server_key_iv: "c2ddbf7efdb6a6f1685c249db942ef42"
server_key_version: "encryptionkey@0"
I20260925 22:53:34.996522 24236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:35.024513 24236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:35.027966 24236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:35.028169 24236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:35.028802 24236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:35.029614 24236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:35.029661 24236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.029727 24236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:35.029758 24236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.038525 24236 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:46277
I20260925 22:53:35.038545 24364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:46277 every 8 connection(s)
I20260925 22:53:35.039414 24236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:35.040325 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24236
I20260925 22:53:35.040458 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:35.040762 4974 external_mini_cluster.cc:1428] Setting key d670015ad7f9c845df90299609e0e844
Sep 25 22:53:35 dist-test-slave-kb1g krb5kdc[24129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.129@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.219.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.130@KRBTEST.COM" created.
I20260925 22:53:35.055155 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.041440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:41771 (local address 127.4.219.190:43215)
0925 22:53:35.041582 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.041585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.042781 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 22:53:35.044225 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.044230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.044294 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.044387 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:35.045649 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.046541 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.047720 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.048165 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.048305 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.051059 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:35.051062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:35.051064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:35.051089 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:35.052679 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.053294 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.053299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.053304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.053391 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.053802 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.053804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.053805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.053991 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:35.054066 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.054768 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.054929 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:53:35.056018 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.041737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43215 (local address 127.4.219.129:41771)
0925 22:53:35.042612 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.042672 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.043965 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.044617 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.044625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.045022 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:35.045498 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.045511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.046686 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.046689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.047593 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.047601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.048326 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.048328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.048471 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.049126 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:35.049142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:35.050889 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:35.052799 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.052804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.052818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.053187 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.053499 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.053502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.053504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.053706 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.054093 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:35.054102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:35.054467 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.054699 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.055058 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":136,"threads_started":1}
I20260925 22:53:35.057608 24365 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43215
I20260925 22:53:35.057884 24365 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:35.058405 24365 heartbeater.cc:511] Master 127.4.219.190:43215 requested a full tablet report, sending...
I20260925 22:53:35.060218 24174 ts_manager.cc:194] Registered new tserver with Master: 864203a208de4990a65c026e0cf32ed8 (127.4.219.129:46277)
I20260925 22:53:35.061928 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.129@KRBTEST.COM'} at 127.4.219.129:41771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
Entry for principal kudu/127.4.219.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab.
I20260925 22:53:35.071122 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:35.182832 24370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:35.183097 24370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:35.183139 24370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:35.187101 24370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:35.187157 24370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:35.187215 24370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:35.192309 24370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.130.keytab
--principal=kudu/127.4.219.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:35.193351 24370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:35.194298 24370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:35.203970 24378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:35.203987 24375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:35.203970 24376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:35.204144 24370 server_base.cc:1034] running on GCE node
I20260925 22:53:35.204790 24370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:35.205438 24370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:35.206647 24370 hybrid_clock.cc:648] HybridClock initialized: now 1790376815206632 us; error 31 us; skew 500 ppm
Sep 25 22:53:35 dist-test-slave-kb1g krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:35.209473 24370 init.cc:377] Logged in from keytab as kudu/127.4.219.130@KRBTEST.COM (short username kudu)
I20260925 22:53:35.210328 24370 webserver.cc:460] Webserver started at http://127.4.219.130:36103/ using document root <none> and password file <none>
I20260925 22:53:35.210894 24370 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:35.210989 24370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:35.211218 24370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:35.213316 24370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "8c098de497e14005b10a5dc8ef3bae30"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "b20f5d16ade9390dd50daef2d8db05f0"
server_key_iv: "b7bcde5d5a62863d0ad018d1d5bf90fc"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.213877 24370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "8c098de497e14005b10a5dc8ef3bae30"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "b20f5d16ade9390dd50daef2d8db05f0"
server_key_iv: "b7bcde5d5a62863d0ad018d1d5bf90fc"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.217553 24370 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:53:35.219890 24385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.221175 24370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:35.221307 24370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "8c098de497e14005b10a5dc8ef3bae30"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "b20f5d16ade9390dd50daef2d8db05f0"
server_key_iv: "b7bcde5d5a62863d0ad018d1d5bf90fc"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.221421 24370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:35.244763 24370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:35.248200 24370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:35.248410 24370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:35.249053 24370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:35.249871 24370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:35.249943 24370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.250015 24370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:35.250061 24370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.258960 24370 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:44523
I20260925 22:53:35.258984 24498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:44523 every 8 connection(s)
I20260925 22:53:35.259835 24370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:35.267653 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24370
I20260925 22:53:35.267800 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:35.268122 4974 external_mini_cluster.cc:1428] Setting key 9825773c87c31327ff2784d8f2f12fda
Sep 25 22:53:35 dist-test-slave-kb1g krb5kdc[24129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.130@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:35.273300 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.261552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:46747 (local address 127.4.219.190:43215)
0925 22:53:35.261690 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.261694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.262467 (+ 773us) server_negotiation.cc:407] Connection header received
0925 22:53:35.263393 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.263404 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.263467 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.263570 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:35.264784 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.265312 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.266464 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.266701 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.266811 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.269523 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:35.269528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:35.269531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:35.269557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:35.271003 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.271511 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.271516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.271517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.271625 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.272062 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.272067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.272070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.272328 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:35.272417 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.273082 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.273184 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:53:35.274559 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.261874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43215 (local address 127.4.219.130:46747)
0925 22:53:35.262317 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.262360 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.263176 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.263743 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.263750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.264105 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:35.264634 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.264649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.265458 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.265463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.266327 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.266335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.266848 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.266853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.267038 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.267576 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:35.267599 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:35.269345 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:35.271132 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.271138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.271152 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.271410 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.271741 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.271746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.271748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.271948 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.272457 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:35.272466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:35.272820 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.273033 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.273393 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260925 22:53:35.275753 24499 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43215
I20260925 22:53:35.276008 24499 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:35.276749 24499 heartbeater.cc:511] Master 127.4.219.190:43215 requested a full tablet report, sending...
I20260925 22:53:35.277890 24174 ts_manager.cc:194] Registered new tserver with Master: 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523)
WARNING: no policy specified for kudu/127.4.219.131@KRBTEST.COM; defaulting to no policy
I20260925 22:53:35.278545 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.130@KRBTEST.COM'} at 127.4.219.130:46747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
Entry for principal kudu/127.4.219.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab.
I20260925 22:53:35.295648 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:35.409749 24504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:35.410017 24504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:35.410059 24504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:35.414156 24504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:35.414219 24504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:35.414320 24504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:35.419741 24504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.131.keytab
--principal=kudu/127.4.219.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:43215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:35.420855 24504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:35.421890 24504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:35.431658 24512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:35.431658 24510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:35.431820 24504 server_base.cc:1034] running on GCE node
W20260925 22:53:35.431672 24509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:35.432371 24504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:35.433048 24504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:35.434201 24504 hybrid_clock.cc:648] HybridClock initialized: now 1790376815434182 us; error 28 us; skew 500 ppm
Sep 25 22:53:35 dist-test-slave-kb1g krb5kdc[24129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:35.436923 24504 init.cc:377] Logged in from keytab as kudu/127.4.219.131@KRBTEST.COM (short username kudu)
I20260925 22:53:35.437744 24504 webserver.cc:460] Webserver started at http://127.4.219.131:42139/ using document root <none> and password file <none>
I20260925 22:53:35.438305 24504 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:35.438398 24504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:35.438630 24504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:35.440754 24504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "d7f1d804cfe349d4be3d9dc71b65930b"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "1a4c6ce46b12366f9755bb32c16a7198"
server_key_iv: "985621aca7c03e15fdc686e7711c3a9f"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.441330 24504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f1d804cfe349d4be3d9dc71b65930b"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "1a4c6ce46b12366f9755bb32c16a7198"
server_key_iv: "985621aca7c03e15fdc686e7711c3a9f"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.444911 24504 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:35.447207 24519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.448385 24504 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:35.448514 24504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "d7f1d804cfe349d4be3d9dc71b65930b"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "1a4c6ce46b12366f9755bb32c16a7198"
server_key_iv: "985621aca7c03e15fdc686e7711c3a9f"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.448690 24504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:35.470804 24504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:35.473724 24504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:35.473945 24504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:35.474545 24504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:35.475371 24504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:35.475443 24504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.475515 24504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:35.475551 24504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.484674 24504 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:33595
I20260925 22:53:35.484736 24632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:33595 every 8 connection(s)
I20260925 22:53:35.485592 24504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:35.492522 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24504
I20260925 22:53:35.492740 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:35.493036 4974 external_mini_cluster.cc:1428] Setting key 306646ce41381c45bd7f9118eb405bb2
Sep 25 22:53:35 dist-test-slave-kb1g krb5kdc[24129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.131@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:35.498736 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.487416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:58443 (local address 127.4.219.190:43215)
0925 22:53:35.487566 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.487570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.488404 (+ 834us) server_negotiation.cc:407] Connection header received
0925 22:53:35.489318 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.489322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.489379 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.489471 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:35.490625 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.491159 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.492282 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.492522 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.492682 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.495492 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:35.495495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:35.495497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:35.495522 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:35.496906 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.497363 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.497365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.497370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.497490 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.497821 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.497823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.497824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.497993 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:35.498065 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.498516 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.498609 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:53:35.499610 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.487795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:43215 (local address 127.4.219.131:58443)
0925 22:53:35.488244 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.488286 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.489113 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.489657 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.489664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.490014 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:35.490478 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.490491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.491285 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.491288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.492157 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.492166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.492726 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.492730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.492944 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.493599 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:53:35.493622 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:35.495329 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:35.497005 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.497010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.497021 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.497265 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.497592 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:53:35.497595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:53:35.497597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:53:35.497740 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:53:35.498078 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:35.498083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:35.498303 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.498466 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.498783 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260925 22:53:35.500782 24633 heartbeater.cc:347] Connected to a master server at 127.4.219.190:43215
I20260925 22:53:35.501045 24633 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:35.501579 24633 heartbeater.cc:511] Master 127.4.219.190:43215 requested a full tablet report, sending...
I20260925 22:53:35.502580 24174 ts_manager.cc:194] Registered new tserver with Master: d7f1d804cfe349d4be3d9dc71b65930b (127.4.219.131:33595)
I20260925 22:53:35.503146 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.219.131@KRBTEST.COM'} at 127.4.219.131:58443
I20260925 22:53:35.507443 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:35.516122 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.509315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.4.219.190:43215)
0925 22:53:35.509432 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.509446 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.509638 (+ 192us) server_negotiation.cc:407] Connection header received
0925 22:53:35.509732 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.509734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.509769 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.509832 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:35.510481 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.511056 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.511815 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.512015 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.512085 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.513380 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:35.513382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:35.513384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:35.513405 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:35.514800 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.515152 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.515154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.515156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.515221 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:35.515432 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:35.515434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:35.515446 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:35.515635 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:35.515728 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.515924 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.516014 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:53:35.518714 24174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:35.521795 24174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:35.536250 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.530038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44523 (local address 127.0.0.1:53802)
0925 22:53:35.530812 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.530825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.530914 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.531565 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.531571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.531607 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.531779 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.531786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533232 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.533235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.534868 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.534881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.535748 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.535917 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.535942 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.536013 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.536081 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":188,"spinlock_wait_cycles":5760,"thread_start_us":79,"threads_started":1}
I20260925 22:53:35.536278 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.530676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.4.219.130:44523)
0925 22:53:35.531159 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.531169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.531189 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:53:35.531253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.531257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.531412 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.531677 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.531887 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533093 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535005 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.535620 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535801 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.535942 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.536039 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.536137 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":111,"threads_started":1}
I20260925 22:53:35.536391 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.530797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.4.219.131:33595)
0925 22:53:35.531456 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.531470 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.531493 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:53:35.531569 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.531576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.531762 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.531915 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.532378 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533289 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.534652 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.535473 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535578 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.535720 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.536159 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.536245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":82,"threads_started":1}
I20260925 22:53:35.536659 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.530537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33595 (local address 127.0.0.1:51990)
0925 22:53:35.531118 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.531131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.531211 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.532086 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.532091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.532114 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.532238 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.532246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533431 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.533445 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.534531 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.534540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.535603 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.535739 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.535812 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.535927 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.536052 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":194,"thread_start_us":83,"threads_started":1}
I20260925 22:53:35.536814 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.530026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.4.219.129:46277)
0925 22:53:35.530953 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.530974 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.530998 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:53:35.531075 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.531079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.531285 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.532064 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.532214 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533299 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535034 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.535847 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.535954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.536093 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.536584 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.536674 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":77,"threads_started":1}
I20260925 22:53:35.536659 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.529870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:46277 (local address 127.0.0.1:40732)
0925 22:53:35.530477 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.530528 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.530658 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.531565 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.531571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.531608 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.531961 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.531967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.533423 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.533429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.534900 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.534909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.536151 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.536154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.536246 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.536276 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.536348 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.536428 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":74,"spinlock_wait_cycles":6656,"thread_start_us":63,"threads_started":1}
I20260925 22:53:35.539913 24433 tablet_service.cc:1467] Processing CreateTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 (DEFAULT_TABLE table=test-table [id=d26c8916cd584f4b91075622d419f508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:35.539911 24299 tablet_service.cc:1467] Processing CreateTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 (DEFAULT_TABLE table=test-table [id=d26c8916cd584f4b91075622d419f508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:35.541080 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 216058d9f7c94f7d99533cfbe3adda98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:35.541080 24433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 216058d9f7c94f7d99533cfbe3adda98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:35.542351 24567 tablet_service.cc:1467] Processing CreateTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 (DEFAULT_TABLE table=test-table [id=d26c8916cd584f4b91075622d419f508]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:35.543264 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 216058d9f7c94f7d99533cfbe3adda98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:35.548422 24650 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Bootstrap starting.
I20260925 22:53:35.549209 24652 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Bootstrap starting.
I20260925 22:53:35.549293 24651 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: Bootstrap starting.
I20260925 22:53:35.550984 24652 tablet_bootstrap.cc:654] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:35.551263 24651 tablet_bootstrap.cc:654] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:35.551697 24650 tablet_bootstrap.cc:654] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:35.552218 24652 log.cc:824] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:35.552723 24650 log.cc:824] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:35.552821 24651 log.cc:824] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:35.554096 24652 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: No bootstrap required, opened a new log
I20260925 22:53:35.554296 24652 ts_tablet_manager.cc:1397] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 22:53:35.554549 24651 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: No bootstrap required, opened a new log
I20260925 22:53:35.554549 24650 tablet_bootstrap.cc:492] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: No bootstrap required, opened a new log
I20260925 22:53:35.554726 24651 ts_tablet_manager.cc:1397] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:53:35.554726 24650 ts_tablet_manager.cc:1397] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:53:35.557225 24652 raft_consensus.cc:348] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.557474 24652 raft_consensus.cc:374] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:35.557546 24652 raft_consensus.cc:729] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f1d804cfe349d4be3d9dc71b65930b, State: Initialized, Role: FOLLOWER
I20260925 22:53:35.557979 24652 consensus_queue.cc:260] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.558441 24651 raft_consensus.cc:348] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.558450 24650 raft_consensus.cc:348] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.558682 24650 raft_consensus.cc:374] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:35.558761 24651 raft_consensus.cc:374] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:35.558794 24650 raft_consensus.cc:729] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c098de497e14005b10a5dc8ef3bae30, State: Initialized, Role: FOLLOWER
I20260925 22:53:35.558851 24651 raft_consensus.cc:729] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864203a208de4990a65c026e0cf32ed8, State: Initialized, Role: FOLLOWER
I20260925 22:53:35.558998 24652 ts_tablet_manager.cc:1428] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260925 22:53:35.559283 24651 consensus_queue.cc:260] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.559384 24633 heartbeater.cc:503] Master 127.4.219.190:43215 was elected leader, sending a full tablet report...
I20260925 22:53:35.559836 24365 heartbeater.cc:503] Master 127.4.219.190:43215 was elected leader, sending a full tablet report...
I20260925 22:53:35.560156 24651 ts_tablet_manager.cc:1428] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:53:35.560271 24650 consensus_queue.cc:260] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.560951 24499 heartbeater.cc:503] Master 127.4.219.190:43215 was elected leader, sending a full tablet report...
I20260925 22:53:35.561264 24650 ts_tablet_manager.cc:1428] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:53:35.605068 24656 raft_consensus.cc:482] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:35.605355 24656 raft_consensus.cc:504] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.605566 24657 raft_consensus.cc:482] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:35.605770 24657 raft_consensus.cc:504] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.606649 24656 leader_election.cc:288] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 864203a208de4990a65c026e0cf32ed8 (127.4.219.129:46277), 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523)
I20260925 22:53:35.607371 24657 leader_election.cc:288] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523), d7f1d804cfe349d4be3d9dc71b65930b (127.4.219.131:33595)
I20260925 22:53:35.611733 24660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:33595 (local address 127.4.219.129:43623)
0925 22:53:35.607484 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.607500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.607601 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.608067 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.608069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.608082 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.608234 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.608241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609207 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.609210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.610431 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.610441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.611297 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.611401 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.611423 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.611494 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.611573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260925 22:53:35.611846 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.606802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:46277 (local address 127.4.219.131:44719)
0925 22:53:35.607086 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.607102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.607192 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.607907 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.607910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.607951 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.608075 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.608082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609220 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.609223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.610430 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.610448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.611432 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611445 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.611549 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.611571 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.611637 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.611700 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"spinlock_wait_cycles":4736}
I20260925 22:53:35.612176 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43623 (local address 127.4.219.131:33595)
0925 22:53:35.607822 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.607826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.607839 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:53:35.607888 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.607902 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.607972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.608049 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.608347 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609093 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.610566 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.611206 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611307 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.611358 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.611994 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.612064 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":31872}
I20260925 22:53:35.612567 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:44719 (local address 127.4.219.129:46277)
0925 22:53:35.607683 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.607687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.607702 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:53:35.607746 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.607749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.607795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.607880 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.608202 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609069 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.610566 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.611219 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611330 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.611377 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.612339 (+ 962us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.612406 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":18816}
I20260925 22:53:35.612751 24587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "864203a208de4990a65c026e0cf32ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" is_pre_election: true
I20260925 22:53:35.612913 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:52283 (local address 127.4.219.130:44523)
0925 22:53:35.607231 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.607237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.608330 (+ 1093us) server_negotiation.cc:407] Connection header received
0925 22:53:35.608497 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.608503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.608568 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.608617 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.609069 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609730 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611556 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.612170 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.612260 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.612310 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.612658 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.612755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":52}
I20260925 22:53:35.613006 24587 raft_consensus.cc:2454] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864203a208de4990a65c026e0cf32ed8 in term 0.
I20260925 22:53:35.613122 24319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864203a208de4990a65c026e0cf32ed8" is_pre_election: true
I20260925 22:53:35.613307 24659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.606906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44523 (local address 127.4.219.131:52283)
0925 22:53:35.608260 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.608273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.608384 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.608750 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.608753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.608772 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.608944 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.608952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609854 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.609857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.611436 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.611443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.612393 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.612397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.612501 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.612549 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.613087 (+ 538us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.613166 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260925 22:53:35.613323 24319 raft_consensus.cc:2454] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f1d804cfe349d4be3d9dc71b65930b in term 0.
I20260925 22:53:35.613489 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:38943 (local address 127.4.219.130:44523)
0925 22:53:35.607788 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.607796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.608012 (+ 216us) server_negotiation.cc:407] Connection header received
0925 22:53:35.608129 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.608132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.608179 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.608252 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:35.608689 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609609 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.611234 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.611920 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.612062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.612107 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.613288 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.613369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260925 22:53:35.613488 24253 leader_election.cc:302] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864203a208de4990a65c026e0cf32ed8, d7f1d804cfe349d4be3d9dc71b65930b; no voters:
I20260925 22:53:35.613718 24661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.607339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:44523 (local address 127.4.219.129:38943)
0925 22:53:35.607831 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.607843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:35.607938 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:35.608372 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:35.608375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:35.608405 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:35.608574 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.608581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.609725 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.609728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.611108 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:35.611119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.612161 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.612164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:35.612262 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.612298 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:53:35.612363 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:53:35.612412 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":52,"thread_start_us":85,"threads_started":1}
I20260925 22:53:35.613744 24657 raft_consensus.cc:2781] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:35.613834 24657 raft_consensus.cc:482] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:35.613796 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c098de497e14005b10a5dc8ef3bae30" is_pre_election: true
I20260925 22:53:35.613878 24657 raft_consensus.cc:3037] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:35.614064 24453 raft_consensus.cc:2454] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f1d804cfe349d4be3d9dc71b65930b in term 0.
I20260925 22:53:35.614303 24521 leader_election.cc:302] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864203a208de4990a65c026e0cf32ed8, d7f1d804cfe349d4be3d9dc71b65930b; no voters:
I20260925 22:53:35.614519 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "864203a208de4990a65c026e0cf32ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c098de497e14005b10a5dc8ef3bae30" is_pre_election: true
I20260925 22:53:35.614530 24656 raft_consensus.cc:2781] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:35.614615 24656 raft_consensus.cc:482] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:35.614620 24453 raft_consensus.cc:2454] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864203a208de4990a65c026e0cf32ed8 in term 0.
I20260925 22:53:35.614655 24656 raft_consensus.cc:3037] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:35.615216 24657 raft_consensus.cc:504] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.615584 24657 leader_election.cc:288] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [CANDIDATE]: Term 1 election: Requested vote from peers 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523), d7f1d804cfe349d4be3d9dc71b65930b (127.4.219.131:33595)
I20260925 22:53:35.615873 24656 raft_consensus.cc:504] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.615887 24587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "864203a208de4990a65c026e0cf32ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f1d804cfe349d4be3d9dc71b65930b"
I20260925 22:53:35.615964 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "864203a208de4990a65c026e0cf32ed8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c098de497e14005b10a5dc8ef3bae30"
I20260925 22:53:35.616078 24453 raft_consensus.cc:3037] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:35.616139 24587 raft_consensus.cc:2379] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 864203a208de4990a65c026e0cf32ed8 in current term 1: Already voted for candidate d7f1d804cfe349d4be3d9dc71b65930b in this term.
I20260925 22:53:35.616253 24656 leader_election.cc:288] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [CANDIDATE]: Term 1 election: Requested vote from peers 864203a208de4990a65c026e0cf32ed8 (127.4.219.129:46277), 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523)
I20260925 22:53:35.616659 24452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c098de497e14005b10a5dc8ef3bae30"
I20260925 22:53:35.616655 24319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "216058d9f7c94f7d99533cfbe3adda98" candidate_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "864203a208de4990a65c026e0cf32ed8"
I20260925 22:53:35.616824 24319 raft_consensus.cc:2379] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d7f1d804cfe349d4be3d9dc71b65930b in current term 1: Already voted for candidate 864203a208de4990a65c026e0cf32ed8 in this term.
I20260925 22:53:35.617187 24521 leader_election.cc:302] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f1d804cfe349d4be3d9dc71b65930b; no voters: 864203a208de4990a65c026e0cf32ed8, 8c098de497e14005b10a5dc8ef3bae30
I20260925 22:53:35.617341 24656 raft_consensus.cc:2726] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 22:53:35.617455 24453 raft_consensus.cc:2454] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864203a208de4990a65c026e0cf32ed8 in term 1.
I20260925 22:53:35.617743 24255 leader_election.cc:302] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 864203a208de4990a65c026e0cf32ed8, 8c098de497e14005b10a5dc8ef3bae30; no voters: d7f1d804cfe349d4be3d9dc71b65930b
I20260925 22:53:35.617900 24657 raft_consensus.cc:2781] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:35.618105 24657 raft_consensus.cc:686] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 864203a208de4990a65c026e0cf32ed8, State: Running, Role: LEADER
I20260925 22:53:35.618407 24657 consensus_queue.cc:237] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } }
I20260925 22:53:35.621575 24172 catalog_manager.cc:5495] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 864203a208de4990a65c026e0cf32ed8 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "864203a208de4990a65c026e0cf32ed8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "864203a208de4990a65c026e0cf32ed8" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 46277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 } health_report { overall_health: UNKNOWN } } }
I20260925 22:53:35.632269 24645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.629034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.4.219.129:46277)
0925 22:53:35.629131 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.629135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.629151 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:53:35.629257 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.629259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.629305 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.629384 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:35.629584 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.630106 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.630928 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.631136 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.631244 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.631552 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:35.631915 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:35.632044 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.632087 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.632133 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 22:53:35.637461 24453 raft_consensus.cc:1261] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 1 FOLLOWER]: Refusing update from remote peer 864203a208de4990a65c026e0cf32ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:35.637465 24587 raft_consensus.cc:1261] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Refusing update from remote peer 864203a208de4990a65c026e0cf32ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:35.638024 24663 consensus_queue.cc:1035] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f1d804cfe349d4be3d9dc71b65930b" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 33595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:35.638185 24657 consensus_queue.cc:1035] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c098de497e14005b10a5dc8ef3bae30" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:35.645098 24668 mvcc.cc:205] Tried to move back new op lower bound from 7333383436844613632 to 7333383436775710720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383436844613632}]
I20260925 22:53:35.647672 24667 mvcc.cc:205] Tried to move back new op lower bound from 7333383436844613632 to 7333383436775710720. Current Snapshot: MvccSnapshot[applied={T|T < 7333383436844613632 or (T in {7333383436844613632})}]
I20260925 22:53:35.652827 24174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:35.652968 24174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:35.655249 24174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fde6462d867e4050bfed6c3eff7fc3c1: Sending DeleteTablet for 3 replicas of tablet 216058d9f7c94f7d99533cfbe3adda98
I20260925 22:53:35.656056 24433 tablet_service.cc:1514] Processing DeleteTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:35 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:53802
I20260925 22:53:35.656049 24567 tablet_service.cc:1514] Processing DeleteTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:35 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:51990
I20260925 22:53:35.656579 24676 tablet_replica.cc:330] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: stopping tablet replica
I20260925 22:53:35.656713 24677 tablet_replica.cc:330] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: stopping tablet replica
I20260925 22:53:35.656713 24299 tablet_service.cc:1514] Processing DeleteTablet for tablet 216058d9f7c94f7d99533cfbe3adda98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:35 UTC) from {username='kudu', principal='kudu/127.4.219.190@KRBTEST.COM'} at 127.0.0.1:40732
I20260925 22:53:35.657016 24677 raft_consensus.cc:2229] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:35.657019 24676 raft_consensus.cc:2229] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:35.657322 24678 tablet_replica.cc:330] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8: stopping tablet replica
I20260925 22:53:35.657321 24677 raft_consensus.cc:2258] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:35.657569 24678 raft_consensus.cc:2229] T 216058d9f7c94f7d99533cfbe3adda98 P 864203a208de4990a65c026e0cf32ed8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:53:35.657873 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24236
I20260925 22:53:35.659349 24677 ts_tablet_manager.cc:1905] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:35.661805 24677 ts_tablet_manager.cc:1918] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:35.661919 24677 log.cc:1196] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/216058d9f7c94f7d99533cfbe3adda98
I20260925 22:53:35.662209 24677 ts_tablet_manager.cc:1939] T 216058d9f7c94f7d99533cfbe3adda98 P 8c098de497e14005b10a5dc8ef3bae30: Deleting consensus metadata
I20260925 22:53:35.663257 24161 catalog_manager.cc:4841] TS 8c098de497e14005b10a5dc8ef3bae30 (127.4.219.130:44523): tablet 216058d9f7c94f7d99533cfbe3adda98 (table test-table [id=d26c8916cd584f4b91075622d419f508]) successfully deleted
I20260925 22:53:35.663359 24676 raft_consensus.cc:2258] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:35.664675 24676 ts_tablet_manager.cc:1905] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:53:35.665753 24159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:46277 (error 108)
I20260925 22:53:35.666801 24644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.666509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:46277 (local address 127.0.0.1:40752)
0925 22:53:35.666616 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:35.666688 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:46277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 22:53:35.666877 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24370
I20260925 22:53:35.667894 24676 ts_tablet_manager.cc:1918] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:35.667999 24676 log.cc:1196] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/216058d9f7c94f7d99533cfbe3adda98
I20260925 22:53:35.668295 24676 ts_tablet_manager.cc:1939] T 216058d9f7c94f7d99533cfbe3adda98 P d7f1d804cfe349d4be3d9dc71b65930b: Deleting consensus metadata
W20260925 22:53:35.669073 24159 catalog_manager.cc:4584] TS 864203a208de4990a65c026e0cf32ed8 (127.4.219.129:46277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 216058d9f7c94f7d99533cfbe3adda98: Network error: Client connection negotiation failed: client connection to 127.4.219.129:46277: connect: Connection refused (error 111)
I20260925 22:53:35.671469 24159 catalog_manager.cc:4841] TS d7f1d804cfe349d4be3d9dc71b65930b (127.4.219.131:33595): tablet 216058d9f7c94f7d99533cfbe3adda98 (table test-table [id=d26c8916cd584f4b91075622d419f508]) successfully deleted
I20260925 22:53:35.673666 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24504
I20260925 22:53:35.679378 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24142
2026-09-25T22:53:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2335 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:35Z Disabled control of system clock
I20260925 22:53:35.709357 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44129
--webserver_interface=127.4.219.190
--webserver_port=0
--builtin_ntp_servers=127.4.219.148:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:35.815346 24683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:35.815599 24683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:35.815634 24683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:35.819489 24683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:35.819537 24683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:35.819553 24683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:35.819569 24683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:35.819583 24683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:35.824112 24683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:35.825100 24683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:35.826354 24683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:35.834352 24688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:35.834359 24689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:35.834367 24691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:35.834522 24683 server_base.cc:1034] running on GCE node
I20260925 22:53:35.835186 24683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:35.836246 24683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:35.837443 24683 hybrid_clock.cc:648] HybridClock initialized: now 1790376815837429 us; error 29 us; skew 500 ppm
I20260925 22:53:35.839164 24683 webserver.cc:460] Webserver started at http://127.4.219.190:42765/ using document root <none> and password file <none>
I20260925 22:53:35.839723 24683 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:35.839812 24683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:35.840013 24683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:35.842031 24683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "4f64e5ac282849d4841732eb8d5d44e0"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "067bf5d74fd13b1be1f1fc0c17248771"
server_key_iv: "8a898cd3db6fe0e46a3dcb024f5dc15c"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.842582 24683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "4f64e5ac282849d4841732eb8d5d44e0"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "067bf5d74fd13b1be1f1fc0c17248771"
server_key_iv: "8a898cd3db6fe0e46a3dcb024f5dc15c"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.846088 24683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:53:35.848287 24697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:35.849488 24683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 22:53:35.849622 24683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "4f64e5ac282849d4841732eb8d5d44e0"
format_stamp: "Formatted at 2026-09-25 22:53:35 on dist-test-slave-kb1g"
server_key: "067bf5d74fd13b1be1f1fc0c17248771"
server_key_iv: "8a898cd3db6fe0e46a3dcb024f5dc15c"
server_key_version: "encryptionkey@0"
I20260925 22:53:35.849733 24683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:35.864812 24683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:35.865469 24683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:35.865662 24683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:35.872195 24683 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44129
I20260925 22:53:35.872210 24749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44129 every 8 connection(s)
I20260925 22:53:35.873149 24683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:35.874487 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24683
I20260925 22:53:35.874634 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:35.874910 4974 external_mini_cluster.cc:1428] Setting key 2c51dffd65fb1131cbdbd6263d0ead5b
I20260925 22:53:35.876494 24750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:35.881769 24750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: Bootstrap starting.
I20260925 22:53:35.884057 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:35.876235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.4.219.190:44129)
0925 22:53:35.876908 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:35.876923 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:35.876963 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:53:35.877615 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:35.877631 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:35.878004 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:35.878323 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:35.878738 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.879656 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.880570 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:35.881028 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:35.881191 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:35.882079 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:35.882085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:35.882100 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:35.882563 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:35.882631 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:35.882658 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:35.882763 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:35.882912 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 22:53:35.883184 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":148,"threads_started":1}
I20260925 22:53:35.884403 24750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:35.885264 24750 log.cc:824] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:35.887161 24750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: No bootstrap required, opened a new log
I20260925 22:53:35.890475 24750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } }
I20260925 22:53:35.890672 24750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:35.890749 24750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f64e5ac282849d4841732eb8d5d44e0, State: Initialized, Role: FOLLOWER
I20260925 22:53:35.891185 24750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } }
I20260925 22:53:35.891327 24750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:35.891395 24750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:35.891492 24750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:35.892508 24750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } }
I20260925 22:53:35.892879 24750 leader_election.cc:302] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f64e5ac282849d4841732eb8d5d44e0; no voters:
I20260925 22:53:35.893193 24750 leader_election.cc:288] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:35.893301 24755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:35.893527 24755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [term 1 LEADER]: Becoming Leader. State: Replica: 4f64e5ac282849d4841732eb8d5d44e0, State: Running, Role: LEADER
I20260925 22:53:35.893826 24755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } }
I20260925 22:53:35.894307 24750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:35.895474 24757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f64e5ac282849d4841732eb8d5d44e0. Latest consensus state: current_term: 1 leader_uuid: "4f64e5ac282849d4841732eb8d5d44e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } } }
I20260925 22:53:35.895598 24757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:35.895840 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f64e5ac282849d4841732eb8d5d44e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f64e5ac282849d4841732eb8d5d44e0" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44129 } } }
I20260925 22:53:35.895944 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:35.896040 24764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:35.898937 24764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:35.903795 24764 catalog_manager.cc:1347] Generated new cluster ID: 536fe6a20f274566bc39ceac55f0de9c
I20260925 22:53:35.903861 24764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:35.916935 24764 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:35.917878 24764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:35.931411 24764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: Generated new TSK 0
I20260925 22:53:35.932135 24764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:35.941308 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.129:0
--local_ip_for_outbound_sockets=127.4.219.129
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--builtin_ntp_servers=127.4.219.148:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:53:36.051091 24774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:36.051330 24774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:36.051366 24774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:36.055545 24774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:36.055603 24774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:36.055689 24774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.129
I20260925 22:53:36.060472 24774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.219.129
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.129
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:36.061568 24774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:36.062916 24774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:36.072270 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.072327 24782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.072304 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:36.072566 24774 server_base.cc:1034] running on GCE node
I20260925 22:53:36.072938 24774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:36.073508 24774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:36.074666 24774 hybrid_clock.cc:648] HybridClock initialized: now 1790376816074650 us; error 26 us; skew 500 ppm
I20260925 22:53:36.076470 24774 webserver.cc:460] Webserver started at http://127.4.219.129:41723/ using document root <none> and password file <none>
I20260925 22:53:36.077075 24774 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:36.077134 24774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:36.077378 24774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:36.079401 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data/instance:
uuid: "529b1e90df2c43b1a6e4a186760c4a38"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "504bbc3be832a44557a812f449712b4c"
server_key_iv: "3bea237f6267c71a36c75463bb9097e4"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.079957 24774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance:
uuid: "529b1e90df2c43b1a6e4a186760c4a38"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "504bbc3be832a44557a812f449712b4c"
server_key_iv: "3bea237f6267c71a36c75463bb9097e4"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.083506 24774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260925 22:53:36.085843 24788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.087015 24774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:36.087153 24774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal
uuid: "529b1e90df2c43b1a6e4a186760c4a38"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "504bbc3be832a44557a812f449712b4c"
server_key_iv: "3bea237f6267c71a36c75463bb9097e4"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.087263 24774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:36.113870 24774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:36.114588 24774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:36.114784 24774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:36.115362 24774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:36.116309 24774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:36.116379 24774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.116458 24774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:36.116510 24774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.125259 24774 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.129:36399
I20260925 22:53:36.125283 24901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.129:36399 every 8 connection(s)
I20260925 22:53:36.126138 24774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/data/info.pb
I20260925 22:53:36.127794 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24774
I20260925 22:53:36.127938 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-0/wal/instance
I20260925 22:53:36.128237 4974 external_mini_cluster.cc:1428] Setting key 7a619611c2188e6f7d8238de635b0166
I20260925 22:53:36.130297 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.130:0
--local_ip_for_outbound_sockets=127.4.219.130
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--builtin_ntp_servers=127.4.219.148:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:36.137071 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.127905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:45515 (local address 127.4.219.190:44129)
0925 22:53:36.128031 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.128035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.129402 (+ 1367us) server_negotiation.cc:407] Connection header received
0925 22:53:36.130995 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.130998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.131053 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.131133 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:36.132332 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.132931 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.134814 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.135099 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.135200 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.136583 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:36.136587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:36.136589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:36.136602 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:36.136622 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:36.136630 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:36.136690 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.136889 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.136956 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:53:36.138399 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.128435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44129 (local address 127.4.219.129:45515)
0925 22:53:36.129242 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.129297 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.130786 (+ 1489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.131332 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.131345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.131476 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:36.132173 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.132194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.133061 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.133067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.134667 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.134683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.135276 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.135282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.135542 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.136206 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:36.136210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:36.136269 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:36.136362 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:36.136719 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:36.136730 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:36.136741 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.136889 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.137236 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":139,"threads_started":1}
I20260925 22:53:36.139652 24902 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44129
I20260925 22:53:36.139917 24902 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:36.140609 24902 heartbeater.cc:511] Master 127.4.219.190:44129 requested a full tablet report, sending...
I20260925 22:53:36.142138 24714 ts_manager.cc:194] Registered new tserver with Master: 529b1e90df2c43b1a6e4a186760c4a38 (127.4.219.129:36399)
I20260925 22:53:36.143234 24714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.129:45515
W20260925 22:53:36.241403 24905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:36.241714 24905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:36.241783 24905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:36.245851 24905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:36.245932 24905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:36.246021 24905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.130
I20260925 22:53:36.250689 24905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.219.130
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.130
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:36.251772 24905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:36.253255 24905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:36.262725 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.262742 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.262753 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:36.262939 24905 server_base.cc:1034] running on GCE node
I20260925 22:53:36.263365 24905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:36.263983 24905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:36.265159 24905 hybrid_clock.cc:648] HybridClock initialized: now 1790376816265141 us; error 29 us; skew 500 ppm
I20260925 22:53:36.266953 24905 webserver.cc:460] Webserver started at http://127.4.219.130:42139/ using document root <none> and password file <none>
I20260925 22:53:36.267504 24905 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:36.267593 24905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:36.267819 24905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:36.269861 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data/instance:
uuid: "1da1ab38875945d9b685e3f40e21bd14"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "6c52c873ba18cab7b15290bf96e464f2"
server_key_iv: "dff6f47d384cef61e8b139c1baff28cf"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.270416 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance:
uuid: "1da1ab38875945d9b685e3f40e21bd14"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "6c52c873ba18cab7b15290bf96e464f2"
server_key_iv: "dff6f47d384cef61e8b139c1baff28cf"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.273840 24905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:36.276116 24919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.277331 24905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:36.277462 24905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal
uuid: "1da1ab38875945d9b685e3f40e21bd14"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "6c52c873ba18cab7b15290bf96e464f2"
server_key_iv: "dff6f47d384cef61e8b139c1baff28cf"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.277575 24905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:36.301246 24905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:36.301967 24905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:36.302167 24905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:36.302752 24905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:36.303673 24905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:36.303743 24905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.303812 24905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:36.303864 24905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.312276 24905 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.130:43401
I20260925 22:53:36.312299 25032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.130:43401 every 8 connection(s)
I20260925 22:53:36.313184 24905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/data/info.pb
I20260925 22:53:36.316859 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 24905
I20260925 22:53:36.317008 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal/instance
I20260925 22:53:36.317328 4974 external_mini_cluster.cc:1428] Setting key 4678e2599032e09d9b78ba95bcce4ed8
I20260925 22:53:36.319594 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.219.131:0
--local_ip_for_outbound_sockets=127.4.219.131
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--builtin_ntp_servers=127.4.219.148:46635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:53:36.324236 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.314986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.130:44431 (local address 127.4.219.190:44129)
0925 22:53:36.315117 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.315121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.316393 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 22:53:36.318380 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.318384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.318442 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.318494 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:36.319740 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.320263 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.322003 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.322418 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.322548 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.323628 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:36.323635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:36.323640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:36.323659 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:36.323688 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:36.323703 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:36.323794 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.323942 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.324025 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:53:36.325058 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.315350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44129 (local address 127.4.219.130:44431)
0925 22:53:36.316196 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.316252 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.318141 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.318912 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.318923 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.319040 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:36.319592 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.319607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.320400 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.320403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.321867 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.321880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.322575 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.322578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.322718 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.323359 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:36.323361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:36.323401 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:36.323471 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:36.323798 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:36.323804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:36.323808 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.323902 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.324123 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":163,"threads_started":1}
I20260925 22:53:36.326181 25033 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44129
I20260925 22:53:36.326406 25033 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:36.326915 25033 heartbeater.cc:511] Master 127.4.219.190:44129 requested a full tablet report, sending...
I20260925 22:53:36.327891 24714 ts_manager.cc:194] Registered new tserver with Master: 1da1ab38875945d9b685e3f40e21bd14 (127.4.219.130:43401)
I20260925 22:53:36.328455 24714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.130:44431
W20260925 22:53:36.436348 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:36.436640 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:36.436707 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:36.440688 25036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:36.440732 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:36.440824 25036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.219.131
I20260925 22:53:36.445453 25036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.219.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.219.131
--webserver_port=0
--tserver_master_addrs=127.4.219.190:44129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.219.131
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:36.446436 25036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:36.447749 25036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:53:36.457198 25036 server_base.cc:1034] running on GCE node
W20260925 22:53:36.457144 25044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.457163 25041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:36.457130 25042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:36.457710 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:36.458303 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:36.459460 25036 hybrid_clock.cc:648] HybridClock initialized: now 1790376816459428 us; error 42 us; skew 500 ppm
I20260925 22:53:36.461303 25036 webserver.cc:460] Webserver started at http://127.4.219.131:41655/ using document root <none> and password file <none>
I20260925 22:53:36.461876 25036 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:36.461938 25036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:36.462185 25036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:36.464285 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data/instance:
uuid: "309d2d380de24e458d653af5002c9777"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "fead5d765557e97e03f402d7bcc91562"
server_key_iv: "4723abcc2e2d9015acaf39ef17cec548"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.464908 25036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance:
uuid: "309d2d380de24e458d653af5002c9777"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "fead5d765557e97e03f402d7bcc91562"
server_key_iv: "4723abcc2e2d9015acaf39ef17cec548"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.468568 25036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:53:36.470885 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.472064 25036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:36.472193 25036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal
uuid: "309d2d380de24e458d653af5002c9777"
format_stamp: "Formatted at 2026-09-25 22:53:36 on dist-test-slave-kb1g"
server_key: "fead5d765557e97e03f402d7bcc91562"
server_key_iv: "4723abcc2e2d9015acaf39ef17cec548"
server_key_version: "encryptionkey@0"
I20260925 22:53:36.472301 25036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:36.483651 25036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:36.484272 25036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:36.484473 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:36.485041 25036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:53:36.485967 25036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:53:36.486035 25036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.486104 25036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:53:36.486152 25036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:36.494850 25036 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.131:36821
I20260925 22:53:36.494910 25163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.131:36821 every 8 connection(s)
I20260925 22:53:36.495934 25036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/data/info.pb
I20260925 22:53:36.496622 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25036
I20260925 22:53:36.496759 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal/instance
I20260925 22:53:36.497041 4974 external_mini_cluster.cc:1428] Setting key d487775c7f7dc35429de28fd96e33f48
I20260925 22:53:36.505256 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.498134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.131:59203 (local address 127.4.219.190:44129)
0925 22:53:36.498283 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.498289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.499509 (+ 1220us) server_negotiation.cc:407] Connection header received
0925 22:53:36.500973 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.500976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.501020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.501093 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:36.501926 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.502438 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.503522 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.503726 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.503799 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.504869 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:36.504871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:36.504873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:36.504888 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:36.504904 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:36.504911 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:36.504964 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.505099 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.505146 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:53:36.506104 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.498648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44129 (local address 127.4.219.131:59203)
0925 22:53:36.499362 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.499413 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.500786 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.501235 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.501242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.501325 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:53:36.501811 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.501823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.502536 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.502538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.503411 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.503419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.503843 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.503846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.503983 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.504643 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:53:36.504645 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:53:36.504683 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:53:36.504745 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:53:36.504978 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:53:36.504984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:53:36.504988 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.505080 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.505288 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260925 22:53:36.507254 25164 heartbeater.cc:347] Connected to a master server at 127.4.219.190:44129
I20260925 22:53:36.507486 25164 heartbeater.cc:464] Registering TS with master...
I20260925 22:53:36.507997 25164 heartbeater.cc:511] Master 127.4.219.190:44129 requested a full tablet report, sending...
I20260925 22:53:36.509130 24714 ts_manager.cc:194] Registered new tserver with Master: 309d2d380de24e458d653af5002c9777 (127.4.219.131:36821)
I20260925 22:53:36.509858 24714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.219.131:59203
I20260925 22:53:36.511544 4974 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:53:36.517036 24753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.513546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.4.219.190:44129)
0925 22:53:36.513684 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.513688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.513877 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:53:36.514020 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.514023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.514065 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.514138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:36.514386 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.514879 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.515654 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.515860 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.515955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.516710 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:36.516713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:36.516715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:36.516724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:36.516739 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:36.516746 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:36.516799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.516891 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.516934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:53:36.519671 24714 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:53:36.523092 24714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:53:36.537304 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.531339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43401 (local address 127.0.0.1:37002)
0925 22:53:36.532114 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.532164 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.532289 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.533326 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.533328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.533345 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:36.533467 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.533474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534850 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.534853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.535954 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.535964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.536799 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.536905 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.536941 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.537038 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.537150 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260925 22:53:36.537313 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.531491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.4.219.130:43401)
0925 22:53:36.532700 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.532711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.532731 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:53:36.532794 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.532798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.533005 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.533165 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:36.533616 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534705 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536084 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.536683 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536800 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.537008 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.537063 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.537149 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":108,"threads_started":1}
I20260925 22:53:36.537777 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.531732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:36821 (local address 127.0.0.1:42694)
0925 22:53:36.532524 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.532537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.532703 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.533362 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.533367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.533387 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:36.533487 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.533492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534443 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.534449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.535805 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.535814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.537140 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.537145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.537282 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.537356 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.537574 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.537651 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":82,"thread_start_us":91,"threads_started":1}
I20260925 22:53:36.538229 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.532121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.4.219.131:36821)
0925 22:53:36.532450 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.532465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.532636 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:53:36.532850 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.532855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.533065 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.533205 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:36.533592 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534325 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.535934 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.536780 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536886 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.537075 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.537955 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.538055 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260925 22:53:36.539099 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.531516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:36399 (local address 127.0.0.1:45202)
0925 22:53:36.532200 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.532213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.532289 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.533171 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.533177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.533199 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:36.533334 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.533342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534814 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.534817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.535991 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.535998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.537165 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.537167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.537287 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.537355 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.537528 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.538975 (+ 1447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":105,"thread_start_us":110,"threads_started":1}
I20260925 22:53:36.539606 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.532003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.4.219.129:36399)
0925 22:53:36.532669 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.532681 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.532708 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:53:36.532791 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.532796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.533005 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.533165 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:36.533471 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.534697 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536114 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.536888 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.536980 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.537119 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.539385 (+ 2266us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.539487 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":113,"threads_started":1}
I20260925 22:53:36.540485 24967 tablet_service.cc:1467] Processing CreateTablet for tablet d593d7df4e2a402d8841933038a83a97 (DEFAULT_TABLE table=test-table [id=b033817bb6434113b7946d132e6b65b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:36.540774 25098 tablet_service.cc:1467] Processing CreateTablet for tablet d593d7df4e2a402d8841933038a83a97 (DEFAULT_TABLE table=test-table [id=b033817bb6434113b7946d132e6b65b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:36.541647 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d593d7df4e2a402d8841933038a83a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:36.541647 25098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d593d7df4e2a402d8841933038a83a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:36.541927 24836 tablet_service.cc:1467] Processing CreateTablet for tablet d593d7df4e2a402d8841933038a83a97 (DEFAULT_TABLE table=test-table [id=b033817bb6434113b7946d132e6b65b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:53:36.542754 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d593d7df4e2a402d8841933038a83a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:36.549191 25181 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Bootstrap starting.
I20260925 22:53:36.549500 25183 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Bootstrap starting.
I20260925 22:53:36.551049 25181 tablet_bootstrap.cc:654] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:36.551090 25183 tablet_bootstrap.cc:654] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:36.551746 25181 log.cc:824] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:36.551746 25183 log.cc:824] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:36.552336 25182 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Bootstrap starting.
I20260925 22:53:36.553740 25183 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: No bootstrap required, opened a new log
I20260925 22:53:36.553948 25183 ts_tablet_manager.cc:1397] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260925 22:53:36.554066 25181 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: No bootstrap required, opened a new log
I20260925 22:53:36.554216 25181 ts_tablet_manager.cc:1397] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 22:53:36.554842 25182 tablet_bootstrap.cc:654] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:36.555608 25182 log.cc:824] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:36.557519 25182 tablet_bootstrap.cc:492] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: No bootstrap required, opened a new log
I20260925 22:53:36.557494 25183 raft_consensus.cc:348] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.557693 25182 ts_tablet_manager.cc:1397] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:53:36.557760 25183 raft_consensus.cc:374] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:36.557813 25183 raft_consensus.cc:729] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1ab38875945d9b685e3f40e21bd14, State: Initialized, Role: FOLLOWER
I20260925 22:53:36.558233 25183 consensus_queue.cc:260] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.558348 25181 raft_consensus.cc:348] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.558593 25181 raft_consensus.cc:374] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:36.558651 25181 raft_consensus.cc:729] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529b1e90df2c43b1a6e4a186760c4a38, State: Initialized, Role: FOLLOWER
I20260925 22:53:36.558959 25033 heartbeater.cc:503] Master 127.4.219.190:44129 was elected leader, sending a full tablet report...
I20260925 22:53:36.559105 25181 consensus_queue.cc:260] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.559208 25183 ts_tablet_manager.cc:1428] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260925 22:53:36.559999 25181 ts_tablet_manager.cc:1428] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:53:36.560376 24902 heartbeater.cc:503] Master 127.4.219.190:44129 was elected leader, sending a full tablet report...
I20260925 22:53:36.561910 25182 raft_consensus.cc:348] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.562139 25182 raft_consensus.cc:374] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:36.562196 25182 raft_consensus.cc:729] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309d2d380de24e458d653af5002c9777, State: Initialized, Role: FOLLOWER
I20260925 22:53:36.562589 25182 consensus_queue.cc:260] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.563506 25182 ts_tablet_manager.cc:1428] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260925 22:53:36.563588 25164 heartbeater.cc:503] Master 127.4.219.190:44129 was elected leader, sending a full tablet report...
W20260925 22:53:36.564085 25034 tablet.cc:2369] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:36.609583 25188 raft_consensus.cc:482] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:53:36.609855 25188 raft_consensus.cc:504] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.611140 25188 leader_election.cc:288] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1ab38875945d9b685e3f40e21bd14 (127.4.219.130:43401), 309d2d380de24e458d653af5002c9777 (127.4.219.131:36821)
I20260925 22:53:36.615183 24904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.611324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43401 (local address 127.4.219.129:38279)
0925 22:53:36.611524 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.611539 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.611644 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.611971 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.611974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.612021 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:36.612159 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.612167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.613106 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.613112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.613963 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.613972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.614761 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.614871 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.614896 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.614968 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.615038 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":21}
I20260925 22:53:36.615237 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.611395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:38279 (local address 127.4.219.130:43401)
0925 22:53:36.611524 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.611530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.611705 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:53:36.611781 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.611788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.611853 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.611941 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:36.612305 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.612979 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614096 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.614632 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614736 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.614776 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.615007 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.615086 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:53:36.615366 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.611536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.219.129:43261 (local address 127.4.219.131:36821)
0925 22:53:36.611696 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.611702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.611845 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:53:36.612004 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.612007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.612051 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.612133 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:53:36.612456 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.612986 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614334 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.614760 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614873 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.614917 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.615164 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.615247 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:53:36.615810 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.611402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.131:36821 (local address 127.4.219.129:43261)
0925 22:53:36.611754 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.611768 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:53:36.611867 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:53:36.612159 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:53:36.612161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:53:36.612176 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:53:36.612335 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.612342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.613206 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.613210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.614195 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:53:36.614215 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.614897 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.614900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:53:36.615030 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.615054 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:53:36.615137 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:53:36.615206 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":20,"thread_start_us":143,"threads_started":1}
I20260925 22:53:36.615824 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d593d7df4e2a402d8841933038a83a97" candidate_uuid: "529b1e90df2c43b1a6e4a186760c4a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1ab38875945d9b685e3f40e21bd14" is_pre_election: true
I20260925 22:53:36.616144 24987 raft_consensus.cc:2454] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529b1e90df2c43b1a6e4a186760c4a38 in term 0.
I20260925 22:53:36.616634 25118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d593d7df4e2a402d8841933038a83a97" candidate_uuid: "529b1e90df2c43b1a6e4a186760c4a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309d2d380de24e458d653af5002c9777" is_pre_election: true
I20260925 22:53:36.616698 24791 leader_election.cc:302] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1ab38875945d9b685e3f40e21bd14, 529b1e90df2c43b1a6e4a186760c4a38; no voters:
I20260925 22:53:36.616945 25118 raft_consensus.cc:2454] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 529b1e90df2c43b1a6e4a186760c4a38 in term 0.
I20260925 22:53:36.616967 25188 raft_consensus.cc:2781] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:53:36.617041 25188 raft_consensus.cc:482] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:53:36.617108 25188 raft_consensus.cc:3037] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:36.618230 25188 raft_consensus.cc:504] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.618630 25188 leader_election.cc:288] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1ab38875945d9b685e3f40e21bd14 (127.4.219.130:43401), 309d2d380de24e458d653af5002c9777 (127.4.219.131:36821)
I20260925 22:53:36.619020 24987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d593d7df4e2a402d8841933038a83a97" candidate_uuid: "529b1e90df2c43b1a6e4a186760c4a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1ab38875945d9b685e3f40e21bd14"
I20260925 22:53:36.619071 25118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d593d7df4e2a402d8841933038a83a97" candidate_uuid: "529b1e90df2c43b1a6e4a186760c4a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309d2d380de24e458d653af5002c9777"
I20260925 22:53:36.619138 24987 raft_consensus.cc:3037] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:36.619217 25118 raft_consensus.cc:3037] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:36.620174 24987 raft_consensus.cc:2454] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529b1e90df2c43b1a6e4a186760c4a38 in term 1.
I20260925 22:53:36.620283 25118 raft_consensus.cc:2454] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 529b1e90df2c43b1a6e4a186760c4a38 in term 1.
I20260925 22:53:36.620493 24791 leader_election.cc:302] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1ab38875945d9b685e3f40e21bd14, 529b1e90df2c43b1a6e4a186760c4a38; no voters:
I20260925 22:53:36.620688 25188 raft_consensus.cc:2781] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:36.620939 25188 raft_consensus.cc:686] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 1 LEADER]: Becoming Leader. State: Replica: 529b1e90df2c43b1a6e4a186760c4a38, State: Running, Role: LEADER
I20260925 22:53:36.621304 25188 consensus_queue.cc:237] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } }
I20260925 22:53:36.624370 24714 catalog_manager.cc:5495] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 529b1e90df2c43b1a6e4a186760c4a38 (127.4.219.129). New cstate: current_term: 1 leader_uuid: "529b1e90df2c43b1a6e4a186760c4a38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "529b1e90df2c43b1a6e4a186760c4a38" member_type: VOTER last_known_addr { host: "127.4.219.129" port: 36399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 } health_report { overall_health: UNKNOWN } } }
W20260925 22:53:36.627224 24903 tablet.cc:2369] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:53:36.635538 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.632026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.4.219.129:36399)
0925 22:53:36.632149 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:36.632155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:36.632225 (+ 70us) server_negotiation.cc:407] Connection header received
0925 22:53:36.632432 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:36.632437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:36.632522 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:36.632670 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:53:36.632960 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.633489 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.634280 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:36.634493 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:36.634564 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:36.634805 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:53:36.635205 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:53:36.635308 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:36.635363 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:53:36.635418 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:53:36.641341 24987 raft_consensus.cc:1261] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 1 FOLLOWER]: Refusing update from remote peer 529b1e90df2c43b1a6e4a186760c4a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:36.641330 25118 raft_consensus.cc:1261] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 1 FOLLOWER]: Refusing update from remote peer 529b1e90df2c43b1a6e4a186760c4a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:53:36.642094 25191 consensus_queue.cc:1035] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309d2d380de24e458d653af5002c9777" member_type: VOTER last_known_addr { host: "127.4.219.131" port: 36821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:36.642277 25188 consensus_queue.cc:1035] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1ab38875945d9b685e3f40e21bd14" member_type: VOTER last_known_addr { host: "127.4.219.130" port: 43401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:53:36.648674 25194 mvcc.cc:205] Tried to move back new op lower bound from 7333383440955310080 to 7333383440883576832. Current Snapshot: MvccSnapshot[applied={T|T < 7333383440955310080}]
I20260925 22:53:36.657644 24714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:36.657821 24714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:53:36.660152 24714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f64e5ac282849d4841732eb8d5d44e0: Sending DeleteTablet for 3 replicas of tablet d593d7df4e2a402d8841933038a83a97
I20260925 22:53:36.660926 24836 tablet_service.cc:1514] Processing DeleteTablet for tablet d593d7df4e2a402d8841933038a83a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:36 UTC) from {username='slave'} at 127.0.0.1:45202
I20260925 22:53:36.660914 24967 tablet_service.cc:1514] Processing DeleteTablet for tablet d593d7df4e2a402d8841933038a83a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:36 UTC) from {username='slave'} at 127.0.0.1:37002
I20260925 22:53:36.661104 25098 tablet_service.cc:1514] Processing DeleteTablet for tablet d593d7df4e2a402d8841933038a83a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:53:36 UTC) from {username='slave'} at 127.0.0.1:42694
I20260925 22:53:36.661407 25204 tablet_replica.cc:330] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: stopping tablet replica
I20260925 22:53:36.661582 25203 tablet_replica.cc:330] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38: stopping tablet replica
I20260925 22:53:36.661825 25205 tablet_replica.cc:330] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: stopping tablet replica
I20260925 22:53:36.661867 25203 raft_consensus.cc:2229] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:53:36.662194 25205 raft_consensus.cc:2229] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:36.662204 25203 raft_consensus.cc:2258] T d593d7df4e2a402d8841933038a83a97 P 529b1e90df2c43b1a6e4a186760c4a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:36.662267 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24774
I20260925 22:53:36.663978 25205 raft_consensus.cc:2258] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:36.664337 25204 raft_consensus.cc:2229] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:53:36.664654 25204 raft_consensus.cc:2258] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:53:36.665565 25205 ts_tablet_manager.cc:1905] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:53:36.665874 25204 ts_tablet_manager.cc:1905] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:53:36.668226 24701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.219.129:36399 (error 108)
I20260925 22:53:36.668377 25205 ts_tablet_manager.cc:1918] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:36.668483 25205 log.cc:1196] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-2/wal/wals/d593d7df4e2a402d8841933038a83a97
I20260925 22:53:36.668507 25204 ts_tablet_manager.cc:1918] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:53:36.668612 25204 log.cc:1196] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Deleting WAL directory at /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376566921023-4974-0/minicluster-data/ts-1/wal/wals/d593d7df4e2a402d8841933038a83a97
I20260925 22:53:36.668831 25205 ts_tablet_manager.cc:1939] T d593d7df4e2a402d8841933038a83a97 P 309d2d380de24e458d653af5002c9777: Deleting consensus metadata
I20260925 22:53:36.668915 25204 ts_tablet_manager.cc:1939] T d593d7df4e2a402d8841933038a83a97 P 1da1ab38875945d9b685e3f40e21bd14: Deleting consensus metadata
I20260925 22:53:36.669042 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24905
I20260925 22:53:36.670267 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.669964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.129:36399 (local address 127.0.0.1:45224)
0925 22:53:36.670091 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.670166 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.129:36399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:53:36.670572 24701 catalog_manager.cc:4584] TS 529b1e90df2c43b1a6e4a186760c4a38 (127.4.219.129:36399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d593d7df4e2a402d8841933038a83a97: Network error: Client connection negotiation failed: client connection to 127.4.219.129:36399: connect: Connection refused (error 111)
I20260925 22:53:36.670768 24698 catalog_manager.cc:4841] TS 309d2d380de24e458d653af5002c9777 (127.4.219.131:36821): tablet d593d7df4e2a402d8841933038a83a97 (table test-table [id=b033817bb6434113b7946d132e6b65b6]) successfully deleted
I20260925 22:53:36.675017 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25036
I20260925 22:53:36.676743 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:36.675176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.130:43401 (local address 127.0.0.1:37008)
0925 22:53:36.676580 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0925 22:53:36.676645 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.130:43401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1288}
I20260925 22:53:36.680385 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 24683
2026-09-25T22:53:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (999 ms)
[----------] 6 tests from EncryptionPolicyTest (10486 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:36 dist-test-slave-kb1g krb5kdc[25209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:36 dist-test-slave-kb1g krb5kdc[25209](info): set up 2 sockets
Sep 25 22:53:36 dist-test-slave-kb1g krb5kdc[25209](info): commencing operation
krb5kdc: starting...
W20260925 22:53:37.708763 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:37 dist-test-slave-kb1g krb5kdc[25209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:37.827203 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:37.934831 25222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:37.935154 25222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:37.935237 25222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:37.939301 25222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:37.939388 25222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:37.939427 25222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:37.939486 25222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:37.939524 25222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:37.944706 25222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:37.945716 25222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:37.946699 25222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:37.954977 25227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:37.954973 25228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:37.955000 25230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:37.955096 25222 server_base.cc:1034] running on GCE node
I20260925 22:53:37.955751 25222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:37.956861 25222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:37.958032 25222 hybrid_clock.cc:648] HybridClock initialized: now 1790376817958000 us; error 43 us; skew 500 ppm
Sep 25 22:53:37 dist-test-slave-kb1g krb5kdc[25209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:37.964013 25222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:37.964947 25222 webserver.cc:460] Webserver started at http://127.0.0.1:37785/ using document root <none> and password file <none>
I20260925 22:53:37.965512 25222 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:37.965608 25222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:37.965848 25222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:37.967869 25222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "c7f348515d754fb7b6c1685ade1abe65"
format_stamp: "Formatted at 2026-09-25 22:53:37 on dist-test-slave-kb1g"
server_key: "c7ad81d38b258f56f197340c5c48c0aa"
server_key_iv: "adef2ca87a1b2cce2eaa02ce817a0869"
server_key_version: "encryptionkey@0"
I20260925 22:53:37.968689 25222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "c7f348515d754fb7b6c1685ade1abe65"
format_stamp: "Formatted at 2026-09-25 22:53:37 on dist-test-slave-kb1g"
server_key: "c7ad81d38b258f56f197340c5c48c0aa"
server_key_iv: "adef2ca87a1b2cce2eaa02ce817a0869"
server_key_version: "encryptionkey@0"
I20260925 22:53:37.972411 25222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:53:37.974792 25237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:37.975977 25222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:37.976121 25222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "c7f348515d754fb7b6c1685ade1abe65"
format_stamp: "Formatted at 2026-09-25 22:53:37 on dist-test-slave-kb1g"
server_key: "c7ad81d38b258f56f197340c5c48c0aa"
server_key_iv: "adef2ca87a1b2cce2eaa02ce817a0869"
server_key_version: "encryptionkey@0"
I20260925 22:53:37.976233 25222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:37.995601 25222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:37.998694 25222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:37.998914 25222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:38.006217 25222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46325
I20260925 22:53:38.006500 25289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46325 every 8 connection(s)
I20260925 22:53:38.007212 25222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:38.009985 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:38.013200 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25222
I20260925 22:53:38.013329 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:38.013622 4974 external_mini_cluster.cc:1428] Setting key ed87abf9a10fa57cdbbd1e267662ea80
I20260925 22:53:38.015991 25290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65: Bootstrap starting.
I20260925 22:53:38.018213 25290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:38.018959 25290 log.cc:824] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:38.021087 25290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65: No bootstrap required, opened a new log
Sep 25 22:53:38 dist-test-slave-kb1g krb5kdc[25209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:53:38.025286 25290 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } }
I20260925 22:53:38.025542 25290 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:38.025609 25290 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f348515d754fb7b6c1685ade1abe65, State: Initialized, Role: FOLLOWER
I20260925 22:53:38.026077 25290 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } }
I20260925 22:53:38.026247 25290 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:38.026326 25290 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:38.026450 25290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:38.027339 25290 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } }
I20260925 22:53:38.027694 25290 leader_election.cc:302] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f348515d754fb7b6c1685ade1abe65; no voters:
I20260925 22:53:38.028043 25290 leader_election.cc:288] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:38.028115 25295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:38.028309 25295 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [term 1 LEADER]: Becoming Leader. State: Replica: c7f348515d754fb7b6c1685ade1abe65, State: Running, Role: LEADER
I20260925 22:53:38.028687 25295 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } }
I20260925 22:53:38.028867 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:38.015126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51577 (local address 127.0.0.1:46325)
0925 22:53:38.015565 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:38.015580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:38.015616 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:53:38.016320 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:38.016339 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:38.016767 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:38.017102 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:38.017907 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:38.018966 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:38.020084 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:38.020598 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:38.020714 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:38.023550 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:38.023555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:38.023569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:38.023596 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:38.025408 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:38.025958 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:38.025963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:38.025967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:38.026075 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:38.026414 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:38.026417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:38.026419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:38.026871 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:38.027094 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:38.027440 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 22:53:38.027800 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260925 22:53:38.029340 25290 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:38.030372 25296 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f348515d754fb7b6c1685ade1abe65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } } }
I20260925 22:53:38.030491 25296 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:38.030777 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f348515d754fb7b6c1685ade1abe65. Latest consensus state: current_term: 1 leader_uuid: "c7f348515d754fb7b6c1685ade1abe65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f348515d754fb7b6c1685ade1abe65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46325 } } }
I20260925 22:53:38.030865 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:38.030882 25304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:38.032289 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:38.034083 25304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:38.040180 25304 catalog_manager.cc:1347] Generated new cluster ID: 70bbf4571ff44ba29cf8411925cfe91e
I20260925 22:53:38.040241 25304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:38.045732 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:38.037459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59385 (local address 127.0.0.1:46325)
0925 22:53:38.037563 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:38.037569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:38.037586 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:53:38.037615 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:38.037617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:38.037679 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:38.037754 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:38.038818 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:38.039594 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:38.040679 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:38.040932 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:38.040990 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:38.042484 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:38.042488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:38.042491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:38.042514 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:38.043955 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:38.044433 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:38.044437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:38.044438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:38.044514 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:38.044862 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:38.044865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:38.044867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:38.045041 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:38.045191 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:38.045459 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 22:53:38.045550 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:53:38.063308 25304 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:38.064253 25304 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:38.072944 25304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f348515d754fb7b6c1685ade1abe65: Generated new TSK 0
I20260925 22:53:38.073669 25304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:38.077989 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25222
W20260925 22:53:38.083455 25207 connection.cc:535] client connection to 127.0.0.1:46325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46325): Connection reset by peer (error 104)
2026-09-25T22:53:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:38 dist-test-slave-kb1g krb5kdc[25321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:38 dist-test-slave-kb1g krb5kdc[25321](info): set up 2 sockets
Sep 25 22:53:38 dist-test-slave-kb1g krb5kdc[25321](info): commencing operation
krb5kdc: starting...
W20260925 22:53:39.108839 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:39 dist-test-slave-kb1g krb5kdc[25321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:39.227335 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:39.338959 25334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:39.339263 25334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:39.339355 25334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:39.343474 25334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:39.343561 25334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:39.343609 25334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:39.343665 25334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:39.343703 25334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:39.349006 25334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:39.350116 25334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:39.351204 25334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:39.359645 25340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:39.359826 25334 server_base.cc:1034] running on GCE node
W20260925 22:53:39.359786 25342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:39.359691 25339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:39.360450 25334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:39.361585 25334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:39.362780 25334 hybrid_clock.cc:648] HybridClock initialized: now 1790376819362761 us; error 25 us; skew 500 ppm
Sep 25 22:53:39 dist-test-slave-kb1g krb5kdc[25321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:39.365691 25334 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:39.366601 25334 webserver.cc:460] Webserver started at http://127.0.0.1:44499/ using document root <none> and password file <none>
I20260925 22:53:39.367187 25334 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:39.367285 25334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:39.367551 25334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:39.369673 25334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "f9e8d1ca02e64c05be93f1ad84b089d4"
format_stamp: "Formatted at 2026-09-25 22:53:39 on dist-test-slave-kb1g"
server_key: "e99b616ca9ac1ee27aa020621d15e6b4"
server_key_iv: "af5939bb5f3b597684ee49bc1526d54e"
server_key_version: "encryptionkey@0"
I20260925 22:53:39.370236 25334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "f9e8d1ca02e64c05be93f1ad84b089d4"
format_stamp: "Formatted at 2026-09-25 22:53:39 on dist-test-slave-kb1g"
server_key: "e99b616ca9ac1ee27aa020621d15e6b4"
server_key_iv: "af5939bb5f3b597684ee49bc1526d54e"
server_key_version: "encryptionkey@0"
I20260925 22:53:39.373981 25334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:39.376358 25349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:39.377614 25334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:39.377765 25334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "f9e8d1ca02e64c05be93f1ad84b089d4"
format_stamp: "Formatted at 2026-09-25 22:53:39 on dist-test-slave-kb1g"
server_key: "e99b616ca9ac1ee27aa020621d15e6b4"
server_key_iv: "af5939bb5f3b597684ee49bc1526d54e"
server_key_version: "encryptionkey@0"
I20260925 22:53:39.377882 25334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:39.393332 25334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:39.396070 25334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:39.396281 25334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:39.403283 25334 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33307
I20260925 22:53:39.403306 25401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33307 every 8 connection(s)
I20260925 22:53:39.404256 25334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:39.407140 25402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:39.412609 25402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4: Bootstrap starting.
I20260925 22:53:39.413704 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25334
I20260925 22:53:39.413858 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:39.414130 4974 external_mini_cluster.cc:1428] Setting key c3b14b46838634c8508a0a48373fcc9e
I20260925 22:53:39.415252 25402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:39.416047 25402 log.cc:824] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:39.418035 25402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4: No bootstrap required, opened a new log
I20260925 22:53:39.421422 25402 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260925 22:53:39.421662 25402 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:39.421744 25402 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e8d1ca02e64c05be93f1ad84b089d4, State: Initialized, Role: FOLLOWER
I20260925 22:53:39.422281 25402 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260925 22:53:39.422470 25402 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:39.422542 25402 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:39.422652 25402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:39.423152 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:39.415269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60901 (local address 127.0.0.1:33307)
0925 22:53:39.415721 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:39.415737 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:39.415771 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:53:39.416584 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:39.416611 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:39.417073 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:39.417419 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:39.417830 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:39.418841 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:39.419711 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:39.420183 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:39.420350 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:39.421284 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:39.421289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:39.421302 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:39.421709 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:39.421775 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:39.421808 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:39.421959 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:39.422101 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:53:39.422241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":149,"threads_started":1}
I20260925 22:53:39.423638 25402 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260925 22:53:39.423977 25402 leader_election.cc:302] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e8d1ca02e64c05be93f1ad84b089d4; no voters:
I20260925 22:53:39.424351 25402 leader_election.cc:288] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:39.424427 25407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:39.424629 25407 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [term 1 LEADER]: Becoming Leader. State: Replica: f9e8d1ca02e64c05be93f1ad84b089d4, State: Running, Role: LEADER
I20260925 22:53:39.424971 25407 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } }
I20260925 22:53:39.425638 25402 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:39.426155 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:39.426852 25409 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e8d1ca02e64c05be93f1ad84b089d4. Latest consensus state: current_term: 1 leader_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } } }
I20260925 22:53:39.426964 25409 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:39.427299 25408 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e8d1ca02e64c05be93f1ad84b089d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33307 } } }
I20260925 22:53:39.427408 25408 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:39.428036 25416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:39.431308 25416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:39.432279 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:39.428438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49429 (local address 127.0.0.1:33307)
0925 22:53:39.428575 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:39.428579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:39.428754 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:53:39.428893 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:39.428896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:39.428936 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:39.429024 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:39.429288 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:39.429791 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:39.430606 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:39.430850 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:39.430958 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:39.431824 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:39.431827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:39.431830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:39.431842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:39.431880 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:39.431891 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:39.431985 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:39.432102 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 22:53:39.432133 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:53:39.437233 25416 catalog_manager.cc:1347] Generated new cluster ID: a02c128c33524424ae8c581cc5ddefdf
I20260925 22:53:39.437345 25416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:39.457345 25416 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:39.458292 25416 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:39.469494 25416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e8d1ca02e64c05be93f1ad84b089d4: Generated new TSK 0
I20260925 22:53:39.470373 25416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:39.479458 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25334
W20260925 22:53:39.484771 25319 connection.cc:535] client connection to 127.0.0.1:33307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33307): Connection reset by peer (error 104)
2026-09-25T22:53:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:39 dist-test-slave-kb1g krb5kdc[25433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:39 dist-test-slave-kb1g krb5kdc[25433](info): set up 2 sockets
Sep 25 22:53:39 dist-test-slave-kb1g krb5kdc[25433](info): commencing operation
krb5kdc: starting...
W20260925 22:53:40.516855 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:40 dist-test-slave-kb1g krb5kdc[25433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:40.633157 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:40.740443 25446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:40.740724 25446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:40.740767 25446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:40.744665 25446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:40.744714 25446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:40.744733 25446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:40.744750 25446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:40.744765 25446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:40.749788 25446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:40.750722 25446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:40.751657 25446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:40.759716 25451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:40.759727 25452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:40.759692 25454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:40.760044 25446 server_base.cc:1034] running on GCE node
I20260925 22:53:40.760818 25446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:40.761845 25446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:40.763002 25446 hybrid_clock.cc:648] HybridClock initialized: now 1790376820762991 us; error 23 us; skew 500 ppm
Sep 25 22:53:40 dist-test-slave-kb1g krb5kdc[25433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:40.765805 25446 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:40.766690 25446 webserver.cc:460] Webserver started at http://127.0.0.1:35641/ using document root <none> and password file <none>
I20260925 22:53:40.767241 25446 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:40.767300 25446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:40.767516 25446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:40.769580 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "0aacca6d2dd64fa0be05e6b069d11c45"
format_stamp: "Formatted at 2026-09-25 22:53:40 on dist-test-slave-kb1g"
server_key: "f106c0afbf6f48d81b98e5dc171a3fa1"
server_key_iv: "e8aa090c5d2d674bbc8107d4839dfb0d"
server_key_version: "encryptionkey@0"
I20260925 22:53:40.770134 25446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "0aacca6d2dd64fa0be05e6b069d11c45"
format_stamp: "Formatted at 2026-09-25 22:53:40 on dist-test-slave-kb1g"
server_key: "f106c0afbf6f48d81b98e5dc171a3fa1"
server_key_iv: "e8aa090c5d2d674bbc8107d4839dfb0d"
server_key_version: "encryptionkey@0"
I20260925 22:53:40.773716 25446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:40.775954 25461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:40.777099 25446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:40.777232 25446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "0aacca6d2dd64fa0be05e6b069d11c45"
format_stamp: "Formatted at 2026-09-25 22:53:40 on dist-test-slave-kb1g"
server_key: "f106c0afbf6f48d81b98e5dc171a3fa1"
server_key_iv: "e8aa090c5d2d674bbc8107d4839dfb0d"
server_key_version: "encryptionkey@0"
I20260925 22:53:40.777344 25446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:40.781713 25446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:40.784850 25446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:40.785048 25446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:40.791719 25446 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41101
I20260925 22:53:40.791745 25513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41101 every 8 connection(s)
I20260925 22:53:40.792660 25446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:40.795202 25514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:40.798554 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25446
I20260925 22:53:40.798712 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:40.798998 4974 external_mini_cluster.cc:1428] Setting key db2cea85954562f231b2cff63d30158b
I20260925 22:53:40.801059 25514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45: Bootstrap starting.
I20260925 22:53:40.803339 25514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:40.804060 25514 log.cc:824] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:40.804327 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:40.800393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59625 (local address 127.0.0.1:41101)
0925 22:53:40.800830 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:40.800843 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:40.800881 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:53:40.801609 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:40.801626 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:40.801925 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:40.802247 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:40.803014 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:40.803019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:40.803031 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:40.803041 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:40.803124 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:40.803144 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:40.803279 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:40.803384 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 22:53:40.803641 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":152,"threads_started":1}
I20260925 22:53:40.805825 25514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45: No bootstrap required, opened a new log
I20260925 22:53:40.807089 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:40.810441 25514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260925 22:53:40.810738 25514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:40.810858 25514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aacca6d2dd64fa0be05e6b069d11c45, State: Initialized, Role: FOLLOWER
I20260925 22:53:40.811018 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:40.809314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52903 (local address 127.0.0.1:41101)
0925 22:53:40.809531 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:40.809537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:40.809712 (+ 175us) server_negotiation.cc:407] Connection header received
0925 22:53:40.809835 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:40.809837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:40.809867 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:40.809922 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:40.810594 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:40.810596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:40.810598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:40.810607 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:40.810629 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:40.810638 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:40.810702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:40.810826 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 22:53:40.810876 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:53:40.811425 25514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260925 22:53:40.811570 25514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:40.811654 25514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:40.811749 25514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:40.812673 25514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260925 22:53:40.812956 25514 leader_election.cc:302] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aacca6d2dd64fa0be05e6b069d11c45; no voters:
I20260925 22:53:40.813181 25514 leader_election.cc:288] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:40.813278 25524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:40.813513 25524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [term 1 LEADER]: Becoming Leader. State: Replica: 0aacca6d2dd64fa0be05e6b069d11c45, State: Running, Role: LEADER
I20260925 22:53:40.813864 25524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } }
I20260925 22:53:40.814378 25514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:40.815457 25526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aacca6d2dd64fa0be05e6b069d11c45. Latest consensus state: current_term: 1 leader_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260925 22:53:40.815572 25526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:40.815934 25533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:40.816884 25525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aacca6d2dd64fa0be05e6b069d11c45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41101 } } }
I20260925 22:53:40.816982 25525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:40.818997 25533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:40.824352 25533 catalog_manager.cc:1347] Generated new cluster ID: 2049f038f89a4d0987edd89100ac4f1b
I20260925 22:53:40.824414 25533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:40.834789 25533 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:40.835124 25533 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:40.844290 25533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aacca6d2dd64fa0be05e6b069d11c45: Generated new TSK 0
I20260925 22:53:40.845101 25533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:40.856253 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25446
2026-09-25T22:53:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:40 dist-test-slave-kb1g krb5kdc[25545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:40 dist-test-slave-kb1g krb5kdc[25545](info): set up 2 sockets
Sep 25 22:53:40 dist-test-slave-kb1g krb5kdc[25545](info): commencing operation
krb5kdc: starting...
W20260925 22:53:41.892712 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:41 dist-test-slave-kb1g krb5kdc[25545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:42.011237 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:42.119632 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:42.119961 25558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:42.120042 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:42.124070 25558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:42.124171 25558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:42.124209 25558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:42.124266 25558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:42.124328 25558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:42.129768 25558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:42.130893 25558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:42.131947 25558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:42.140300 25564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:42.140283 25563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:42.140283 25566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:42.140413 25558 server_base.cc:1034] running on GCE node
I20260925 22:53:42.141163 25558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:42.142333 25558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:42.143522 25558 hybrid_clock.cc:648] HybridClock initialized: now 1790376822143503 us; error 31 us; skew 500 ppm
Sep 25 22:53:42 dist-test-slave-kb1g krb5kdc[25545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:42.146404 25558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:42.147406 25558 webserver.cc:460] Webserver started at http://127.0.0.1:40801/ using document root <none> and password file <none>
I20260925 22:53:42.147979 25558 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:42.148072 25558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:42.148308 25558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:42.150427 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "3267969d112d4e539054b484dc264b08"
format_stamp: "Formatted at 2026-09-25 22:53:42 on dist-test-slave-kb1g"
server_key: "0f643ae50e5cbd1656429c804055c263"
server_key_iv: "28d341fc798f6fddc795a0a05bbf6a62"
server_key_version: "encryptionkey@0"
I20260925 22:53:42.150995 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "3267969d112d4e539054b484dc264b08"
format_stamp: "Formatted at 2026-09-25 22:53:42 on dist-test-slave-kb1g"
server_key: "0f643ae50e5cbd1656429c804055c263"
server_key_iv: "28d341fc798f6fddc795a0a05bbf6a62"
server_key_version: "encryptionkey@0"
I20260925 22:53:42.154639 25558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:53:42.156981 25573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:42.158124 25558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:42.158263 25558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "3267969d112d4e539054b484dc264b08"
format_stamp: "Formatted at 2026-09-25 22:53:42 on dist-test-slave-kb1g"
server_key: "0f643ae50e5cbd1656429c804055c263"
server_key_iv: "28d341fc798f6fddc795a0a05bbf6a62"
server_key_version: "encryptionkey@0"
I20260925 22:53:42.158375 25558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:42.189687 25558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:42.193104 25558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:42.193320 25558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:42.200284 25558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44849
I20260925 22:53:42.200318 25625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44849 every 8 connection(s)
I20260925 22:53:42.201290 25558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:42.203907 25626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:42.207612 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25558
I20260925 22:53:42.207753 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:42.208020 4974 external_mini_cluster.cc:1428] Setting key 254e10cf2476973c7c68b6aa6a7fe849
I20260925 22:53:42.209642 25626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08: Bootstrap starting.
I20260925 22:53:42.211714 25626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:42.212571 25626 log.cc:824] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:42.214380 25626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08: No bootstrap required, opened a new log
Sep 25 22:53:42 dist-test-slave-kb1g krb5kdc[25545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:53:42.218217 25626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260925 22:53:42.218425 25626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:42.218504 25626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3267969d112d4e539054b484dc264b08, State: Initialized, Role: FOLLOWER
I20260925 22:53:42.218967 25626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260925 22:53:42.219086 25626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:42.219168 25626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:42.219277 25626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:42.220211 25626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260925 22:53:42.220602 25626 leader_election.cc:302] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3267969d112d4e539054b484dc264b08; no voters:
I20260925 22:53:42.220937 25626 leader_election.cc:288] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:42.221040 25631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:42.221246 25631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [term 1 LEADER]: Becoming Leader. State: Replica: 3267969d112d4e539054b484dc264b08, State: Running, Role: LEADER
I20260925 22:53:42.221616 25631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } }
I20260925 22:53:42.222245 25626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:42.222879 25631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3267969d112d4e539054b484dc264b08. Latest consensus state: current_term: 1 leader_uuid: "3267969d112d4e539054b484dc264b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } } }
I20260925 22:53:42.222987 25631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:42.223119 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:42.209386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58441 (local address 127.0.0.1:44849)
0925 22:53:42.209836 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:42.209861 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:42.209897 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:53:42.210514 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:42.210530 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:42.210863 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:42.211198 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:42.212024 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:42.213408 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:42.214301 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:42.214834 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:42.215002 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:42.217829 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:42.217834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:42.217846 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:42.217872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:42.219830 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:42.220361 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:42.220366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:42.220372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:42.220490 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:42.220807 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:42.220810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:42.220812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:42.221386 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:42.221570 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:42.221878 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 22:53:42.222167 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":120,"threads_started":1}
I20260925 22:53:42.223428 25632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3267969d112d4e539054b484dc264b08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3267969d112d4e539054b484dc264b08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44849 } } }
I20260925 22:53:42.223593 25632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:42.223874 25638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:42.226300 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:42.227293 25638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:42.233057 25638 catalog_manager.cc:1347] Generated new cluster ID: 15106f0699414767bcae85301336aca7
I20260925 22:53:42.233157 25638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:42.237762 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:42.228432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44111 (local address 127.0.0.1:44849)
0925 22:53:42.228558 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:42.228563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:42.228802 (+ 239us) server_negotiation.cc:407] Connection header received
0925 22:53:42.228993 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:42.228997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:42.229042 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:42.229092 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:42.230033 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:42.230815 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:42.231986 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:42.232323 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:42.232439 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:42.234106 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:42.234110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:42.234113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:42.234136 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:42.235864 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:42.236364 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:42.236368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:42.236370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:42.236453 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:42.236798 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:42.236802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:42.236804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:42.237027 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:42.237129 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:42.237488 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 22:53:42.237600 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260925 22:53:42.238655 25638 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:42.239622 25638 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:42.245337 25638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3267969d112d4e539054b484dc264b08: Generated new TSK 0
I20260925 22:53:42.246133 25638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:42.250998 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25558
2026-09-25T22:53:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:42 dist-test-slave-kb1g krb5kdc[25657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:42 dist-test-slave-kb1g krb5kdc[25657](info): set up 2 sockets
Sep 25 22:53:42 dist-test-slave-kb1g krb5kdc[25657](info): commencing operation
krb5kdc: starting...
W20260925 22:53:43.292798 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:43 dist-test-slave-kb1g krb5kdc[25657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:43.415403 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:43.525416 25670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:43.525766 25670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:43.525858 25670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:43.529930 25670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:43.530027 25670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:43.530066 25670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:43.530126 25670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:43.530164 25670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:43.535607 25670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:43.536834 25670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:43.537897 25670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:43.546205 25676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:43.546223 25675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:43.546231 25678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:43.546571 25670 server_base.cc:1034] running on GCE node
I20260925 22:53:43.547098 25670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:43.548105 25670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:43.549255 25670 hybrid_clock.cc:648] HybridClock initialized: now 1790376823549237 us; error 28 us; skew 500 ppm
Sep 25 22:53:43 dist-test-slave-kb1g krb5kdc[25657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:43.561463 25670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:43.562405 25670 webserver.cc:460] Webserver started at http://127.0.0.1:45371/ using document root <none> and password file <none>
I20260925 22:53:43.562978 25670 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:43.563037 25670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:43.563263 25670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:43.565423 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "abbebe53bd214fb280583eebfd65dceb"
format_stamp: "Formatted at 2026-09-25 22:53:43 on dist-test-slave-kb1g"
server_key: "3f10d3486390393a34e5d56d6295e286"
server_key_iv: "d7fafc5f8a80359c63f655026677fcd7"
server_key_version: "encryptionkey@0"
I20260925 22:53:43.565979 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "abbebe53bd214fb280583eebfd65dceb"
format_stamp: "Formatted at 2026-09-25 22:53:43 on dist-test-slave-kb1g"
server_key: "3f10d3486390393a34e5d56d6295e286"
server_key_iv: "d7fafc5f8a80359c63f655026677fcd7"
server_key_version: "encryptionkey@0"
I20260925 22:53:43.569624 25670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:53:43.572075 25685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:43.573369 25670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:43.573474 25670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "abbebe53bd214fb280583eebfd65dceb"
format_stamp: "Formatted at 2026-09-25 22:53:43 on dist-test-slave-kb1g"
server_key: "3f10d3486390393a34e5d56d6295e286"
server_key_iv: "d7fafc5f8a80359c63f655026677fcd7"
server_key_version: "encryptionkey@0"
I20260925 22:53:43.573598 25670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:43.584941 25670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:43.588093 25670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:43.588280 25670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:43.595212 25670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39207
I20260925 22:53:43.595229 25737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39207 every 8 connection(s)
I20260925 22:53:43.596139 25670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:43.598865 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:43.601446 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25670
I20260925 22:53:43.601600 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:43.601910 4974 external_mini_cluster.cc:1428] Setting key 153af96249ba13101ecfff4748bfc8ac
I20260925 22:53:43.605124 25738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: Bootstrap starting.
I20260925 22:53:43.607494 25738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:43.608319 25738 log.cc:824] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:43.610474 25738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: No bootstrap required, opened a new log
I20260925 22:53:43.611140 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:43.603357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32965 (local address 127.0.0.1:39207)
0925 22:53:43.603785 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:43.603799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:43.603839 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:53:43.604471 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:43.604488 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:43.604905 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:43.605237 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:43.605645 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:43.606649 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:43.607904 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:43.608374 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:43.608587 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:43.609475 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:43.609480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:43.609493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:43.609869 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:43.609920 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:43.609942 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:43.610016 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:43.610176 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 22:53:43.610405 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":133,"threads_started":1}
I20260925 22:53:43.614074 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:43.614432 25738 raft_consensus.cc:348] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260925 22:53:43.614713 25738 raft_consensus.cc:374] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:43.614784 25738 raft_consensus.cc:729] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbebe53bd214fb280583eebfd65dceb, State: Initialized, Role: FOLLOWER
I20260925 22:53:43.615250 25738 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260925 22:53:43.615484 25738 raft_consensus.cc:388] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:43.615599 25738 raft_consensus.cc:482] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:43.615703 25738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:43.616966 25738 raft_consensus.cc:504] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260925 22:53:43.617338 25738 leader_election.cc:302] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbebe53bd214fb280583eebfd65dceb; no voters:
I20260925 22:53:43.617688 25738 leader_election.cc:288] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:43.617740 25748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:43.617949 25748 raft_consensus.cc:686] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [term 1 LEADER]: Becoming Leader. State: Replica: abbebe53bd214fb280583eebfd65dceb, State: Running, Role: LEADER
I20260925 22:53:43.618289 25748 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } }
I20260925 22:53:43.618980 25738 sys_catalog.cc:564] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:43.619423 25748 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbebe53bd214fb280583eebfd65dceb. Latest consensus state: current_term: 1 leader_uuid: "abbebe53bd214fb280583eebfd65dceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } } }
I20260925 22:53:43.620152 25748 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:43.620043 25749 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbebe53bd214fb280583eebfd65dceb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbebe53bd214fb280583eebfd65dceb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39207 } } }
I20260925 22:53:43.620464 25749 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:43.620630 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:43.616284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50295 (local address 127.0.0.1:39207)
0925 22:53:43.616401 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:43.616405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:43.616672 (+ 267us) server_negotiation.cc:407] Connection header received
0925 22:53:43.616803 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:43.616815 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:43.616854 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:43.616915 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:43.617222 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:43.617783 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:43.618761 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:43.619046 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:43.619140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:43.620014 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:43.620018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:43.620021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:43.620033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:43.620060 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:43.620072 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:43.620133 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:43.620434 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 22:53:43.620482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 22:53:43.623510 25764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:43.623574 25764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:43.623652 25757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:43.624388 25757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:43.629796 25757 catalog_manager.cc:1347] Generated new cluster ID: bd3318d9461f447288aa93e53c467c48
I20260925 22:53:43.629869 25757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:43.638214 25757 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:43.639151 25757 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:43.646409 25757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abbebe53bd214fb280583eebfd65dceb: Generated new TSK 0
I20260925 22:53:43.647104 25757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:43.653117 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25670
2026-09-25T22:53:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:43 dist-test-slave-kb1g krb5kdc[25769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:43 dist-test-slave-kb1g krb5kdc[25769](info): set up 2 sockets
Sep 25 22:53:43 dist-test-slave-kb1g krb5kdc[25769](info): commencing operation
krb5kdc: starting...
W20260925 22:53:44.753212 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:44 dist-test-slave-kb1g krb5kdc[25769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:44.871959 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:44.980381 25782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:44.980661 25782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:44.980703 25782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:44.984579 25782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:44.984630 25782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:44.984649 25782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:44.984665 25782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:44.984680 25782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:44.989823 25782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:44.990796 25782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:44.991745 25782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:45.000115 25787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:45.000174 25782 server_base.cc:1034] running on GCE node
W20260925 22:53:45.000115 25790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:45.000116 25788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:45.000990 25782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:45.002089 25782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:45.003269 25782 hybrid_clock.cc:648] HybridClock initialized: now 1790376825003238 us; error 44 us; skew 500 ppm
Sep 25 22:53:45 dist-test-slave-kb1g krb5kdc[25769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:45.005986 25782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:45.006872 25782 webserver.cc:460] Webserver started at http://127.0.0.1:37621/ using document root <none> and password file <none>
I20260925 22:53:45.007489 25782 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:45.007580 25782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:45.007820 25782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:45.009949 25782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "8c29649426ed4426a9f2388f64029b06"
format_stamp: "Formatted at 2026-09-25 22:53:45 on dist-test-slave-kb1g"
server_key: "ed0d213eae4e3244de511d9faf9590cd"
server_key_iv: "2ec5cf6ce20848c71fa39aa3a934edb3"
server_key_version: "encryptionkey@0"
I20260925 22:53:45.010537 25782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "8c29649426ed4426a9f2388f64029b06"
format_stamp: "Formatted at 2026-09-25 22:53:45 on dist-test-slave-kb1g"
server_key: "ed0d213eae4e3244de511d9faf9590cd"
server_key_iv: "2ec5cf6ce20848c71fa39aa3a934edb3"
server_key_version: "encryptionkey@0"
I20260925 22:53:45.014339 25782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:53:45.016773 25797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:45.017997 25782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:45.018141 25782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "8c29649426ed4426a9f2388f64029b06"
format_stamp: "Formatted at 2026-09-25 22:53:45 on dist-test-slave-kb1g"
server_key: "ed0d213eae4e3244de511d9faf9590cd"
server_key_iv: "2ec5cf6ce20848c71fa39aa3a934edb3"
server_key_version: "encryptionkey@0"
I20260925 22:53:45.018263 25782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:45.022933 25782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:45.025797 25782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:45.025982 25782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:45.032522 25782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38235
I20260925 22:53:45.032608 25849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38235 every 8 connection(s)
I20260925 22:53:45.033473 25782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:45.036073 25850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:45.037778 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25782
I20260925 22:53:45.037935 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:45.038197 4974 external_mini_cluster.cc:1428] Setting key c7270b148464186ef47b37b585bfbae7
I20260925 22:53:45.042666 25850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06: Bootstrap starting.
I20260925 22:53:45.043529 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:45.039555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35115 (local address 127.0.0.1:38235)
0925 22:53:45.039967 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:45.039986 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:45.040026 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:53:45.040843 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:45.040867 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:45.041211 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:45.041561 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:45.042251 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:45.042257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:45.042273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:45.042284 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:45.042346 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:45.042372 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:45.042503 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:45.042569 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:53:45.042709 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260925 22:53:45.045104 25850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:45.045990 25850 log.cc:824] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:45.046361 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:45.047960 25850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06: No bootstrap required, opened a new log
I20260925 22:53:45.050036 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:45.048370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38047 (local address 127.0.0.1:38235)
0925 22:53:45.048482 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:45.048486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:45.048758 (+ 272us) server_negotiation.cc:407] Connection header received
0925 22:53:45.048888 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:45.048891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:45.048920 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:45.048983 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:45.049653 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:45.049656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:45.049658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:45.049668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:45.049691 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:45.049701 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:45.049780 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:45.049869 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:53:45.049899 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:53:45.052405 25850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:53:45.052703 25850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:45.052773 25850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c29649426ed4426a9f2388f64029b06, State: Initialized, Role: FOLLOWER
I20260925 22:53:45.053169 25850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:53:45.053321 25850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:45.053385 25850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:45.053476 25850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:45.054350 25850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:53:45.054684 25850 leader_election.cc:302] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c29649426ed4426a9f2388f64029b06; no voters:
I20260925 22:53:45.054979 25850 leader_election.cc:288] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:45.055133 25860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:45.055348 25860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [term 1 LEADER]: Becoming Leader. State: Replica: 8c29649426ed4426a9f2388f64029b06, State: Running, Role: LEADER
I20260925 22:53:45.055691 25860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } }
I20260925 22:53:45.056259 25850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:45.057566 25861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c29649426ed4426a9f2388f64029b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260925 22:53:45.057683 25861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:45.057952 25862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c29649426ed4426a9f2388f64029b06. Latest consensus state: current_term: 1 leader_uuid: "8c29649426ed4426a9f2388f64029b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c29649426ed4426a9f2388f64029b06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38235 } } }
I20260925 22:53:45.058117 25862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:45.058427 25869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:45.061067 25869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:45.066254 25869 catalog_manager.cc:1347] Generated new cluster ID: aa0742d3a085472b9c5050189956ac87
I20260925 22:53:45.066334 25869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:45.080693 25869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:45.081094 25869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:45.095439 25869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c29649426ed4426a9f2388f64029b06: Generated new TSK 0
I20260925 22:53:45.096146 25869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:45.100660 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25782
2026-09-25T22:53:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:45 dist-test-slave-kb1g krb5kdc[25881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:45 dist-test-slave-kb1g krb5kdc[25881](info): set up 2 sockets
Sep 25 22:53:45 dist-test-slave-kb1g krb5kdc[25881](info): commencing operation
krb5kdc: starting...
W20260925 22:53:46.136821 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:46.255503 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:46.366114 25894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:46.366407 25894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:46.366492 25894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:46.370482 25894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:46.370560 25894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:46.370597 25894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:46.370656 25894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:46.370692 25894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:46.376291 25894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:46.377429 25894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:46.378468 25894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:46.386701 25900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:46.386785 25894 server_base.cc:1034] running on GCE node
W20260925 22:53:46.386793 25899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:46.386790 25902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:46.387513 25894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:46.388504 25894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:46.389714 25894 hybrid_clock.cc:648] HybridClock initialized: now 1790376826389699 us; error 25 us; skew 500 ppm
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:46.392287 25894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:46.393170 25894 webserver.cc:460] Webserver started at http://127.0.0.1:43963/ using document root <none> and password file <none>
I20260925 22:53:46.393711 25894 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:46.393810 25894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:46.394028 25894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:46.396020 25894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "5d82deea72664cee9967de691ed63fde"
format_stamp: "Formatted at 2026-09-25 22:53:46 on dist-test-slave-kb1g"
server_key: "a1a3c34ef382c54072db3ca1b023e892"
server_key_iv: "d2ed1019acc0a00ee6eb9383c5336b04"
server_key_version: "encryptionkey@0"
I20260925 22:53:46.396639 25894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "5d82deea72664cee9967de691ed63fde"
format_stamp: "Formatted at 2026-09-25 22:53:46 on dist-test-slave-kb1g"
server_key: "a1a3c34ef382c54072db3ca1b023e892"
server_key_iv: "d2ed1019acc0a00ee6eb9383c5336b04"
server_key_version: "encryptionkey@0"
I20260925 22:53:46.400163 25894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:53:46.402683 25909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:46.403880 25894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:46.404017 25894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "5d82deea72664cee9967de691ed63fde"
format_stamp: "Formatted at 2026-09-25 22:53:46 on dist-test-slave-kb1g"
server_key: "a1a3c34ef382c54072db3ca1b023e892"
server_key_iv: "d2ed1019acc0a00ee6eb9383c5336b04"
server_key_version: "encryptionkey@0"
I20260925 22:53:46.404130 25894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:46.434211 25894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:46.437317 25894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:46.437521 25894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:46.444308 25894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40915
I20260925 22:53:46.444382 25961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40915 every 8 connection(s)
I20260925 22:53:46.445330 25894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:46.448088 25962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:46.451674 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 25894
I20260925 22:53:46.451828 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:46.452099 4974 external_mini_cluster.cc:1428] Setting key 8b89e964d9a8ef6a58f1168b9a09c2b8
I20260925 22:53:46.454188 25962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde: Bootstrap starting.
I20260925 22:53:46.456904 25962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:46.457773 25962 log.cc:824] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:46.459908 25962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde: No bootstrap required, opened a new log
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:53:46.464224 25962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260925 22:53:46.464496 25962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:46.464608 25962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d82deea72664cee9967de691ed63fde, State: Initialized, Role: FOLLOWER
I20260925 22:53:46.465114 25962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260925 22:53:46.465276 25962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:46.465365 25962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:46.465464 25962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:46.466427 25962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260925 22:53:46.466790 25962 leader_election.cc:302] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d82deea72664cee9967de691ed63fde; no voters:
I20260925 22:53:46.467116 25962 leader_election.cc:288] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:46.467231 25967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:46.467393 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:46.453562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:59529 (local address 127.0.0.1:40915)
0925 22:53:46.454008 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:46.454027 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:46.454071 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:53:46.454895 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:46.454918 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:46.455355 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:46.455740 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:46.456380 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:46.457928 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:46.458770 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:46.459420 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:46.459576 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:46.462037 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:46.462044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:46.462060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:46.462091 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:46.464090 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:46.464616 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:46.464622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:46.464626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:46.464793 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:46.465125 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:46.465129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:46.465130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:46.465658 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:46.465908 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:46.466208 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:53:46.466433 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260925 22:53:46.467470 25967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [term 1 LEADER]: Becoming Leader. State: Replica: 5d82deea72664cee9967de691ed63fde, State: Running, Role: LEADER
I20260925 22:53:46.467980 25967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } }
I20260925 22:53:46.468500 25962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:46.469699 25968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d82deea72664cee9967de691ed63fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260925 22:53:46.470094 25968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:46.470822 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:46.469993 25969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d82deea72664cee9967de691ed63fde. Latest consensus state: current_term: 1 leader_uuid: "5d82deea72664cee9967de691ed63fde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d82deea72664cee9967de691ed63fde" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40915 } } }
I20260925 22:53:46.470999 25969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:46.471331 25982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:46.474936 25982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:46.481465 25982 catalog_manager.cc:1347] Generated new cluster ID: 72520f41ad39456d85c148628489ad32
I20260925 22:53:46.481479 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:46.472967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:46617 (local address 127.0.0.1:40915)
0925 22:53:46.473108 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:46.473113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:46.473364 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:53:46.473508 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:46.473511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:46.473563 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:46.473647 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:46.474319 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:46.474844 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:46.475710 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:46.475977 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:46.476087 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:46.477835 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:46.477839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:46.477842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:46.477868 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:46.479677 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:46.480128 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:46.480132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:46.480136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:46.480226 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:46.480614 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:46.480619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:46.480625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:46.480797 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:46.480950 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:46.481212 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 22:53:46.481308 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:53:46.481540 25982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:46.496740 25982 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:46.497980 25982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:46.511546 25982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d82deea72664cee9967de691ed63fde: Generated new TSK 0
I20260925 22:53:46.512363 25982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:46.525170 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 25894
W20260925 22:53:46.530460 25879 connection.cc:535] client connection to 127.0.0.1:40915 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40915): Connection reset by peer (error 104)
2026-09-25T22:53:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25993](info): set up 2 sockets
Sep 25 22:53:46 dist-test-slave-kb1g krb5kdc[25993](info): commencing operation
krb5kdc: starting...
W20260925 22:53:47.560883 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:47 dist-test-slave-kb1g krb5kdc[25993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:47.683382 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:47.801967 26006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:47.802228 26006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:47.802269 26006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:47.806207 26006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:47.806264 26006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:47.806284 26006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:47.806299 26006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:47.806314 26006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:47.811437 26006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:47.812381 26006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:47.813382 26006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:47.821281 26014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:47.821285 26012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:47.821281 26011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:47.821660 26006 server_base.cc:1034] running on GCE node
I20260925 22:53:47.822206 26006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:47.823228 26006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:47.824410 26006 hybrid_clock.cc:648] HybridClock initialized: now 1790376827824380 us; error 41 us; skew 500 ppm
Sep 25 22:53:47 dist-test-slave-kb1g krb5kdc[25993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:47.827204 26006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:47.828096 26006 webserver.cc:460] Webserver started at http://127.0.0.1:33401/ using document root <none> and password file <none>
I20260925 22:53:47.828735 26006 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:47.828830 26006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:47.829077 26006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:47.831084 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "f5d87e5a39e84af0bfde71977c0b026b"
format_stamp: "Formatted at 2026-09-25 22:53:47 on dist-test-slave-kb1g"
server_key: "1ce24b2d505cc5c99c8d98f1eceb0b42"
server_key_iv: "2e572aed805a51303b3b94e0463aced5"
server_key_version: "encryptionkey@0"
I20260925 22:53:47.831652 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "f5d87e5a39e84af0bfde71977c0b026b"
format_stamp: "Formatted at 2026-09-25 22:53:47 on dist-test-slave-kb1g"
server_key: "1ce24b2d505cc5c99c8d98f1eceb0b42"
server_key_iv: "2e572aed805a51303b3b94e0463aced5"
server_key_version: "encryptionkey@0"
I20260925 22:53:47.835433 26006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:53:47.838014 26021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:47.839309 26006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:47.839473 26006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "f5d87e5a39e84af0bfde71977c0b026b"
format_stamp: "Formatted at 2026-09-25 22:53:47 on dist-test-slave-kb1g"
server_key: "1ce24b2d505cc5c99c8d98f1eceb0b42"
server_key_iv: "2e572aed805a51303b3b94e0463aced5"
server_key_version: "encryptionkey@0"
I20260925 22:53:47.839607 26006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:47.862510 26006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:47.865509 26006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:47.865713 26006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:47.872735 26006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40943
I20260925 22:53:47.872758 26073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40943 every 8 connection(s)
I20260925 22:53:47.873699 26006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:47.876343 26074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:47.879885 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26006
I20260925 22:53:47.880028 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:47.880307 4974 external_mini_cluster.cc:1428] Setting key 36c861077a76efe3b6a7b2dbc6c12168
I20260925 22:53:47.882169 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b: Bootstrap starting.
I20260925 22:53:47.884349 26074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:47.885164 26074 log.cc:824] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:47.886977 26074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b: No bootstrap required, opened a new log
I20260925 22:53:47.889179 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:47.881704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:43535 (local address 127.0.0.1:40943)
0925 22:53:47.882142 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:47.882156 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:47.882192 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:53:47.882871 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:47.882888 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:47.883221 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:47.883575 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:47.883979 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:47.885068 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:47.886008 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:47.886482 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:47.886650 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:47.887545 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:47.887556 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:47.887567 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:47.887933 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:47.887981 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:47.888002 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:47.888137 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:47.888323 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:53:47.888460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260925 22:53:47.890653 26074 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } }
I20260925 22:53:47.890921 26074 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:47.891006 26074 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d87e5a39e84af0bfde71977c0b026b, State: Initialized, Role: FOLLOWER
I20260925 22:53:47.891459 26074 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } }
I20260925 22:53:47.891659 26074 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:47.891734 26074 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:47.891852 26074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:47.892292 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:47.892923 26074 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } }
I20260925 22:53:47.893266 26074 leader_election.cc:302] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d87e5a39e84af0bfde71977c0b026b; no voters:
I20260925 22:53:47.893592 26074 leader_election.cc:288] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:47.893672 26081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:47.893913 26081 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [term 1 LEADER]: Becoming Leader. State: Replica: f5d87e5a39e84af0bfde71977c0b026b, State: Running, Role: LEADER
I20260925 22:53:47.894730 26081 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } }
I20260925 22:53:47.894924 26074 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:47.896377 26084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d87e5a39e84af0bfde71977c0b026b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } } }
I20260925 22:53:47.896504 26084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:47.896374 26085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d87e5a39e84af0bfde71977c0b026b. Latest consensus state: current_term: 1 leader_uuid: "f5d87e5a39e84af0bfde71977c0b026b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d87e5a39e84af0bfde71977c0b026b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40943 } } }
I20260925 22:53:47.896831 26085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:47.896888 26096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:47.900233 26096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:47.900446 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:47.894414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:34107 (local address 127.0.0.1:40943)
0925 22:53:47.894546 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:47.894551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:47.895960 (+ 1409us) server_negotiation.cc:407] Connection header received
0925 22:53:47.896152 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:47.896156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:47.896209 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:47.896262 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:47.896773 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:47.897518 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:47.898620 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:47.898943 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:47.899047 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:47.899951 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:47.899954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:47.899957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:47.899970 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:47.899997 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:47.900010 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:47.900099 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:47.900279 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 22:53:47.900315 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:53:47.905885 26096 catalog_manager.cc:1347] Generated new cluster ID: f4649b9c0fb043c6b903937c575bb7f2
I20260925 22:53:47.905995 26096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:47.926635 26096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:47.927928 26096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:47.934028 26096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d87e5a39e84af0bfde71977c0b026b: Generated new TSK 0
I20260925 22:53:47.934729 26096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:47.945194 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26006
W20260925 22:53:47.951339 25991 connection.cc:535] client connection to 127.0.0.1:40943 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40943): Connection reset by peer (error 104)
2026-09-25T22:53:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:48 dist-test-slave-kb1g krb5kdc[26105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:48 dist-test-slave-kb1g krb5kdc[26105](info): set up 2 sockets
Sep 25 22:53:48 dist-test-slave-kb1g krb5kdc[26105](info): commencing operation
krb5kdc: starting...
W20260925 22:53:49.024695 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:49 dist-test-slave-kb1g krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:49.144326 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:49.253402 26118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:49.253659 26118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:49.253700 26118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:49.257632 26118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:49.257687 26118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:49.257706 26118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:49.257722 26118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:49.257737 26118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:49.263015 26118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:49.264039 26118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:49.265071 26118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:49.273290 26124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:49.273439 26118 server_base.cc:1034] running on GCE node
W20260925 22:53:49.273312 26126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:49.273376 26123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:49.274080 26118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:49.275126 26118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:49.276293 26118 hybrid_clock.cc:648] HybridClock initialized: now 1790376829276258 us; error 46 us; skew 500 ppm
Sep 25 22:53:49 dist-test-slave-kb1g krb5kdc[26105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:49.284161 26118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:49.285230 26118 webserver.cc:460] Webserver started at http://127.0.0.1:32989/ using document root <none> and password file <none>
I20260925 22:53:49.285825 26118 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:49.285923 26118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:49.286161 26118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:49.288249 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "02c5e58f9ca6427586e00506c35bb70a"
format_stamp: "Formatted at 2026-09-25 22:53:49 on dist-test-slave-kb1g"
server_key: "5c4d0a851683da2ca7ab383bfe430987"
server_key_iv: "abec9645c95b19c545df40900def1be1"
server_key_version: "encryptionkey@0"
I20260925 22:53:49.288830 26118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "02c5e58f9ca6427586e00506c35bb70a"
format_stamp: "Formatted at 2026-09-25 22:53:49 on dist-test-slave-kb1g"
server_key: "5c4d0a851683da2ca7ab383bfe430987"
server_key_iv: "abec9645c95b19c545df40900def1be1"
server_key_version: "encryptionkey@0"
I20260925 22:53:49.292598 26118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:53:49.295105 26133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:49.296332 26118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:49.296512 26118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "02c5e58f9ca6427586e00506c35bb70a"
format_stamp: "Formatted at 2026-09-25 22:53:49 on dist-test-slave-kb1g"
server_key: "5c4d0a851683da2ca7ab383bfe430987"
server_key_iv: "abec9645c95b19c545df40900def1be1"
server_key_version: "encryptionkey@0"
I20260925 22:53:49.296692 26118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:49.301613 26118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:49.304488 26118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:49.304721 26118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:49.311787 26118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46873
I20260925 22:53:49.311800 26185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46873 every 8 connection(s)
I20260925 22:53:49.312803 26118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:49.315462 26186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:49.320354 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26118
I20260925 22:53:49.320500 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:49.320798 4974 external_mini_cluster.cc:1428] Setting key 766720af3ca9f0068d811211d46923ad
I20260925 22:53:49.321182 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: Bootstrap starting.
I20260925 22:53:49.323652 26186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:49.324441 26186 log.cc:824] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:49.326011 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:49.322209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:57979 (local address 127.0.0.1:46873)
0925 22:53:49.322658 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:49.322673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:49.322707 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:53:49.323332 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:49.323349 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:49.323634 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:49.324027 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:49.324848 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:49.324853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:49.324865 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:49.324875 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:49.324925 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:49.324944 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:49.325054 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:49.325178 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 22:53:49.325346 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":136,"threads_started":1}
I20260925 22:53:49.326431 26186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: No bootstrap required, opened a new log
I20260925 22:53:49.328859 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:49.330431 26186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260925 22:53:49.330678 26186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:49.330763 26186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c5e58f9ca6427586e00506c35bb70a, State: Initialized, Role: FOLLOWER
I20260925 22:53:49.331255 26186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260925 22:53:49.331434 26186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:49.331508 26186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:49.331630 26186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:49.332420 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:49.330827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:49781 (local address 127.0.0.1:46873)
0925 22:53:49.330936 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:49.330940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:49.331145 (+ 205us) server_negotiation.cc:407] Connection header received
0925 22:53:49.331240 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:49.331243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:49.331282 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:49.331361 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:49.332080 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:49.332083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:49.332086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:49.332097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:49.332123 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:49.332135 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:49.332227 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:49.332268 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 22:53:49.332306 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:53:49.332927 26186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260925 22:53:49.333294 26186 leader_election.cc:302] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02c5e58f9ca6427586e00506c35bb70a; no voters:
I20260925 22:53:49.333611 26186 leader_election.cc:288] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:49.333716 26196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:49.333935 26196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [term 1 LEADER]: Becoming Leader. State: Replica: 02c5e58f9ca6427586e00506c35bb70a, State: Running, Role: LEADER
I20260925 22:53:49.334302 26196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } }
I20260925 22:53:49.334868 26186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:49.335988 26197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02c5e58f9ca6427586e00506c35bb70a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } } }
I20260925 22:53:49.336112 26197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:49.336488 26198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c5e58f9ca6427586e00506c35bb70a. Latest consensus state: current_term: 1 leader_uuid: "02c5e58f9ca6427586e00506c35bb70a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c5e58f9ca6427586e00506c35bb70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46873 } } }
I20260925 22:53:49.336630 26198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:49.339432 26211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:49.339495 26211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:49.339574 26205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:49.340238 26205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:49.345829 26205 catalog_manager.cc:1347] Generated new cluster ID: 6b54c1f1994945509662e3e1299f64b8
I20260925 22:53:49.345906 26205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:49.381356 26205 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:49.381748 26205 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:49.392495 26205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02c5e58f9ca6427586e00506c35bb70a: Generated new TSK 0
I20260925 22:53:49.393416 26205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:49.398972 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26118
2026-09-25T22:53:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:49 dist-test-slave-kb1g krb5kdc[26217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:49 dist-test-slave-kb1g krb5kdc[26217](info): set up 2 sockets
Sep 25 22:53:49 dist-test-slave-kb1g krb5kdc[26217](info): commencing operation
krb5kdc: starting...
W20260925 22:53:50.452973 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:50.575002 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:50.684669 26230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:50.684948 26230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:50.685001 26230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:50.689385 26230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:50.689452 26230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:50.689472 26230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:50.689531 26230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:50.689569 26230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:50.694959 26230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:50.696070 26230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:50.697124 26230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:50.705675 26236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:50.705667 26235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:50.705688 26238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:50.706274 26230 server_base.cc:1034] running on GCE node
I20260925 22:53:50.706830 26230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:50.707967 26230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:50.709148 26230 hybrid_clock.cc:648] HybridClock initialized: now 1790376830709112 us; error 51 us; skew 500 ppm
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:50.712114 26230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:50.713121 26230 webserver.cc:460] Webserver started at http://127.0.0.1:35597/ using document root <none> and password file <none>
I20260925 22:53:50.713697 26230 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:50.713789 26230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:50.714013 26230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:50.716039 26230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "bb38b2a45a5347eca053792109dac88a"
format_stamp: "Formatted at 2026-09-25 22:53:50 on dist-test-slave-kb1g"
server_key: "64cc5a7bad1818d8979bdcfc6809ab6c"
server_key_iv: "3f5f5b3b90356c75b201472e8ddec402"
server_key_version: "encryptionkey@0"
I20260925 22:53:50.716662 26230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "bb38b2a45a5347eca053792109dac88a"
format_stamp: "Formatted at 2026-09-25 22:53:50 on dist-test-slave-kb1g"
server_key: "64cc5a7bad1818d8979bdcfc6809ab6c"
server_key_iv: "3f5f5b3b90356c75b201472e8ddec402"
server_key_version: "encryptionkey@0"
I20260925 22:53:50.720309 26230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:53:50.722682 26245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:50.723829 26230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:50.723961 26230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "bb38b2a45a5347eca053792109dac88a"
format_stamp: "Formatted at 2026-09-25 22:53:50 on dist-test-slave-kb1g"
server_key: "64cc5a7bad1818d8979bdcfc6809ab6c"
server_key_iv: "3f5f5b3b90356c75b201472e8ddec402"
server_key_version: "encryptionkey@0"
I20260925 22:53:50.724068 26230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:50.737706 26230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:50.740311 26230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:50.740504 26230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:50.747291 26230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45793
I20260925 22:53:50.747318 26297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45793 every 8 connection(s)
I20260925 22:53:50.748235 26230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:50.750942 26298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:50.751116 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26230
I20260925 22:53:50.751253 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:50.751551 4974 external_mini_cluster.cc:1428] Setting key 4ee67051873232f2bdb1f6d642238146
I20260925 22:53:50.757321 26298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a: Bootstrap starting.
I20260925 22:53:50.759552 26298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a: Neither blocks nor log segments found. Creating new log.
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:53:50.760258 26298 log.cc:824] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:50.762686 26298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a: No bootstrap required, opened a new log
I20260925 22:53:50.766211 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:50.752989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:57605 (local address 127.0.0.1:45793)
0925 22:53:50.753409 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:50.753424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:50.753459 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:53:50.754250 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:50.754268 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:50.754606 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:50.754954 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:50.755742 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:50.756774 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:50.757677 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:50.758137 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:50.758303 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:50.760963 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:50.760967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:50.760980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:50.761025 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:50.762961 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:50.763459 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:50.763465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:50.763471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:50.763610 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:50.763914 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:50.763918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:50.763919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:50.764407 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:50.764641 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:50.764973 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 22:53:50.765249 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":135,"threads_started":1}
I20260925 22:53:50.767345 26298 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260925 22:53:50.767629 26298 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:50.767689 26298 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb38b2a45a5347eca053792109dac88a, State: Initialized, Role: FOLLOWER
I20260925 22:53:50.768302 26298 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260925 22:53:50.768479 26298 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:50.768564 26298 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:50.768669 26298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:50.769399 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:50.769788 26298 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260925 22:53:50.770140 26298 leader_election.cc:302] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb38b2a45a5347eca053792109dac88a; no voters:
I20260925 22:53:50.770462 26298 leader_election.cc:288] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:50.770562 26303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:50.770771 26303 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [term 1 LEADER]: Becoming Leader. State: Replica: bb38b2a45a5347eca053792109dac88a, State: Running, Role: LEADER
I20260925 22:53:50.771116 26303 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } }
I20260925 22:53:50.771818 26298 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:50.772766 26308 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb38b2a45a5347eca053792109dac88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } } }
I20260925 22:53:50.772888 26308 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:50.773118 26308 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb38b2a45a5347eca053792109dac88a. Latest consensus state: current_term: 1 leader_uuid: "bb38b2a45a5347eca053792109dac88a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb38b2a45a5347eca053792109dac88a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45793 } } }
I20260925 22:53:50.773191 26308 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:50.773540 26317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:50.776770 26317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:50.780701 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:50.771707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:56673 (local address 127.0.0.1:45793)
0925 22:53:50.771809 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:50.771813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:50.771829 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:53:50.771929 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:50.771933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:50.771977 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:50.772024 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:50.773058 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:50.773824 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:50.774954 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:50.775292 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:50.775405 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:50.777140 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:50.777144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:50.777147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:50.777170 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:50.778864 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:50.779379 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:50.779382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:50.779384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:50.779468 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:50.779793 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:50.779796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:50.779798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:50.779979 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:50.780128 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:50.780395 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 22:53:50.780484 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:53:50.783073 26317 catalog_manager.cc:1347] Generated new cluster ID: 4852a357930140458c42bd4d5fc9fd3a
I20260925 22:53:50.783143 26317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:50.799429 26317 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:50.800356 26317 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:50.808580 26317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb38b2a45a5347eca053792109dac88a: Generated new TSK 0
I20260925 22:53:50.809237 26317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:50.818099 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26230
W20260925 22:53:50.823364 26215 connection.cc:535] client connection to 127.0.0.1:45793 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45793): Connection reset by peer (error 104)
2026-09-25T22:53:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26329](info): set up 2 sockets
Sep 25 22:53:50 dist-test-slave-kb1g krb5kdc[26329](info): commencing operation
krb5kdc: starting...
W20260925 22:53:51.868695 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:51 dist-test-slave-kb1g krb5kdc[26329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:51.993597 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:52.103137 26342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:52.103439 26342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:52.103483 26342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:52.107499 26342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:52.107574 26342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:52.107594 26342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:52.107612 26342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:52.107627 26342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:52.113138 26342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:52.114241 26342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:52.115288 26342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:52.124275 26348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:52.124301 26350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:52.124320 26347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:52.124353 26342 server_base.cc:1034] running on GCE node
I20260925 22:53:52.125144 26342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:52.126282 26342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:52.127466 26342 hybrid_clock.cc:648] HybridClock initialized: now 1790376832127453 us; error 27 us; skew 500 ppm
Sep 25 22:53:52 dist-test-slave-kb1g krb5kdc[26329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:52.130373 26342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:52.131374 26342 webserver.cc:460] Webserver started at http://127.0.0.1:34541/ using document root <none> and password file <none>
I20260925 22:53:52.131975 26342 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:52.132072 26342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:52.132319 26342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:52.134500 26342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139"
format_stamp: "Formatted at 2026-09-25 22:53:52 on dist-test-slave-kb1g"
server_key: "cff935d1ebf1f56ff59a5a85596c59d5"
server_key_iv: "8ee9a83dc59c0d5a95e71df7458b030d"
server_key_version: "encryptionkey@0"
I20260925 22:53:52.135174 26342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139"
format_stamp: "Formatted at 2026-09-25 22:53:52 on dist-test-slave-kb1g"
server_key: "cff935d1ebf1f56ff59a5a85596c59d5"
server_key_iv: "8ee9a83dc59c0d5a95e71df7458b030d"
server_key_version: "encryptionkey@0"
I20260925 22:53:52.139133 26342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:53:52.141821 26357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:52.143203 26342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:53:52.143373 26342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139"
format_stamp: "Formatted at 2026-09-25 22:53:52 on dist-test-slave-kb1g"
server_key: "cff935d1ebf1f56ff59a5a85596c59d5"
server_key_iv: "8ee9a83dc59c0d5a95e71df7458b030d"
server_key_version: "encryptionkey@0"
I20260925 22:53:52.143528 26342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:52.158631 26342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:52.162036 26342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:52.162271 26342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:52.169826 26342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45709
I20260925 22:53:52.169853 26409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45709 every 8 connection(s)
I20260925 22:53:52.170833 26342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:52.173655 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:52.179551 26410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: Bootstrap starting.
I20260925 22:53:52.180187 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26342
I20260925 22:53:52.180326 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:52.180645 4974 external_mini_cluster.cc:1428] Setting key e5d31ffbc1dbdf45dfb070af734673ff
I20260925 22:53:52.182618 26410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:52.183547 26410 log.cc:824] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:52.185909 26410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: No bootstrap required, opened a new log
I20260925 22:53:52.190567 26410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } }
I20260925 22:53:52.190886 26410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:52.190934 26410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa7ea2f4f384c79bbbf45e3bfffb139, State: Initialized, Role: FOLLOWER
I20260925 22:53:52.191443 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:52.182124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:45095 (local address 127.0.0.1:45709)
0925 22:53:52.182619 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:52.182640 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:52.182687 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:53:52.183627 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:52.183656 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:52.184115 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:52.184496 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:52.184951 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:52.186376 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:52.187308 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:52.188048 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:52.188215 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:52.189151 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:52.189159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:52.189181 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:52.189755 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:52.189825 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:52.189857 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:52.190005 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:52.190248 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 22:53:52.190397 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":134,"threads_started":1}
I20260925 22:53:52.191457 26410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } }
I20260925 22:53:52.191627 26410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:52.191704 26410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:52.191804 26410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:52.193008 26410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } }
I20260925 22:53:52.193404 26410 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fa7ea2f4f384c79bbbf45e3bfffb139; no voters:
I20260925 22:53:52.193792 26410 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:52.193897 26415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:52.194178 26415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa7ea2f4f384c79bbbf45e3bfffb139, State: Running, Role: LEADER
I20260925 22:53:52.194545 26415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } }
I20260925 22:53:52.194885 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:52.195206 26410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:52.197606 26416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } } }
I20260925 22:53:52.197825 26416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:52.198307 26417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa7ea2f4f384c79bbbf45e3bfffb139. Latest consensus state: current_term: 1 leader_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa7ea2f4f384c79bbbf45e3bfffb139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45709 } } }
I20260925 22:53:52.198410 26417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139 [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:52.200766 26433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:52.200850 26433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:52.200917 26435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:52.201807 26435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:52.203310 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:52.198090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:52405 (local address 127.0.0.1:45709)
0925 22:53:52.198199 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:52.198205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:52.198807 (+ 602us) server_negotiation.cc:407] Connection header received
0925 22:53:52.198931 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:52.198933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:52.198975 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:52.199046 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:52.199391 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:52.200177 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:52.201354 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:52.201669 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:52.201778 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:52.202795 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:52.202799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:52.202802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:52.202816 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:52.202845 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:52.202858 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:52.202941 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:52.203121 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 22:53:52.203159 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:53:52.208227 26435 catalog_manager.cc:1347] Generated new cluster ID: b6732fdfda8c4f62aabbe15cb741784e
I20260925 22:53:52.208359 26435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:52.215982 26435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:52.217365 26435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:52.230928 26435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fa7ea2f4f384c79bbbf45e3bfffb139: Generated new TSK 0
I20260925 22:53:52.231912 26435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:52.244988 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26342
W20260925 22:53:52.251120 26327 connection.cc:535] client connection to 127.0.0.1:45709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45709): Connection reset by peer (error 104)
2026-09-25T22:53:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:52 dist-test-slave-kb1g krb5kdc[26441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:52 dist-test-slave-kb1g krb5kdc[26441](info): set up 2 sockets
Sep 25 22:53:52 dist-test-slave-kb1g krb5kdc[26441](info): commencing operation
krb5kdc: starting...
W20260925 22:53:53.301250 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:53 dist-test-slave-kb1g krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:53:53.427657 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:53.539119 26454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:53.539395 26454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:53.539467 26454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:53.543583 26454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:53.543665 26454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:53.543685 26454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:53.543702 26454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:53.543752 26454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:53.549154 26454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:53.550304 26454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:53.551404 26454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:53.560145 26460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:53.560279 26454 server_base.cc:1034] running on GCE node
W20260925 22:53:53.560145 26462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:53.560191 26459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:53.561079 26454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:53.562244 26454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:53.563460 26454 hybrid_clock.cc:648] HybridClock initialized: now 1790376833563445 us; error 28 us; skew 500 ppm
Sep 25 22:53:53 dist-test-slave-kb1g krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:53.566423 26454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:53:53.567307 26454 webserver.cc:460] Webserver started at http://127.0.0.1:39377/ using document root <none> and password file <none>
I20260925 22:53:53.567828 26454 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:53.567885 26454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:53.568048 26454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:53.570120 26454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "fbb92fd6b66e4d3db34fc973fe93adcf"
format_stamp: "Formatted at 2026-09-25 22:53:53 on dist-test-slave-kb1g"
server_key: "318efb2499e33dab9c455a393b9e970e"
server_key_iv: "55e663fec59dcf2cc7c486ccdf0abe1f"
server_key_version: "encryptionkey@0"
I20260925 22:53:53.570626 26454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "fbb92fd6b66e4d3db34fc973fe93adcf"
format_stamp: "Formatted at 2026-09-25 22:53:53 on dist-test-slave-kb1g"
server_key: "318efb2499e33dab9c455a393b9e970e"
server_key_iv: "55e663fec59dcf2cc7c486ccdf0abe1f"
server_key_version: "encryptionkey@0"
I20260925 22:53:53.574211 26454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 22:53:53.576490 26469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:53.577922 26454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:53.578032 26454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "fbb92fd6b66e4d3db34fc973fe93adcf"
format_stamp: "Formatted at 2026-09-25 22:53:53 on dist-test-slave-kb1g"
server_key: "318efb2499e33dab9c455a393b9e970e"
server_key_iv: "55e663fec59dcf2cc7c486ccdf0abe1f"
server_key_version: "encryptionkey@0"
I20260925 22:53:53.578130 26454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:53.583076 26454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:53.586331 26454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:53.586493 26454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:53.593623 26454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38309
I20260925 22:53:53.593652 26521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38309 every 8 connection(s)
I20260925 22:53:53.594682 26454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:53.597590 26522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:53.603277 26522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf: Bootstrap starting.
I20260925 22:53:53.604467 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26454
I20260925 22:53:53.604781 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:53.605213 4974 external_mini_cluster.cc:1428] Setting key 1ba4d10eb3c91781b66f701311b4bd24
I20260925 22:53:53.606235 26522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:53.607055 26522 log.cc:824] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:53.609405 26522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf: No bootstrap required, opened a new log
I20260925 22:53:53.610543 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:53.606371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:34159 (local address 127.0.0.1:38309)
0925 22:53:53.606833 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:53.606848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:53.606883 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:53:53.607770 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:53.607798 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:53.608130 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:53.608425 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:53.609292 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:53.609297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:53.609308 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:53.609318 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:53.609369 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:53.609388 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:53.609496 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:53.609638 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 22:53:53.609824 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":147,"threads_started":1}
I20260925 22:53:53.613724 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:53.613884 26522 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260925 22:53:53.614133 26522 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:53.614208 26522 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb92fd6b66e4d3db34fc973fe93adcf, State: Initialized, Role: FOLLOWER
I20260925 22:53:53.614653 26522 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260925 22:53:53.614773 26522 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:53.614856 26522 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:53.614975 26522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:53.616089 26522 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260925 22:53:53.616495 26522 leader_election.cc:302] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbb92fd6b66e4d3db34fc973fe93adcf; no voters:
I20260925 22:53:53.616900 26522 leader_election.cc:288] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:53.617022 26532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:53.617246 26532 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [term 1 LEADER]: Becoming Leader. State: Replica: fbb92fd6b66e4d3db34fc973fe93adcf, State: Running, Role: LEADER
I20260925 22:53:53.617651 26532 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } }
I20260925 22:53:53.618166 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:53.616178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:35647 (local address 127.0.0.1:38309)
0925 22:53:53.616302 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:53.616307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:53.616522 (+ 215us) server_negotiation.cc:407] Connection header received
0925 22:53:53.616655 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:53.616659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:53.616703 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:53.616778 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:53.617557 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:53.617561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:53.617563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:53.617574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:53.617601 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:53.617614 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:53.617709 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:53.618009 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:53:53.618054 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:53:53.618276 26522 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:53.619109 26532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbb92fd6b66e4d3db34fc973fe93adcf. Latest consensus state: current_term: 1 leader_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } } }
I20260925 22:53:53.619287 26532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:53.619551 26534 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbb92fd6b66e4d3db34fc973fe93adcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38309 } } }
I20260925 22:53:53.619647 26534 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:53.619709 26541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:53.623576 26541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:53.629541 26541 catalog_manager.cc:1347] Generated new cluster ID: 3c7af93a08a94fc8a9db6eb00d536083
I20260925 22:53:53.629614 26541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:53.640827 26541 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:53.641197 26541 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:53.654677 26541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbb92fd6b66e4d3db34fc973fe93adcf: Generated new TSK 0
I20260925 22:53:53.655596 26541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:53.665040 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26454
2026-09-25T22:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:53 dist-test-slave-kb1g krb5kdc[26553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:53 dist-test-slave-kb1g krb5kdc[26553](info): set up 2 sockets
Sep 25 22:53:53 dist-test-slave-kb1g krb5kdc[26553](info): commencing operation
krb5kdc: starting...
W20260925 22:53:54.696851 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:54 dist-test-slave-kb1g krb5kdc[26553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:54.814842 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36963
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:54.924574 26566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:54.924832 26566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:54.924873 26566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:54.928802 26566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:54.928853 26566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:54.928871 26566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:54.928891 26566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:54.928906 26566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:54.934031 26566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:36963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:54.935009 26566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:54.935979 26566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:54.944906 26571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:54.944906 26572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:54.944906 26574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:54.945159 26566 server_base.cc:1034] running on GCE node
I20260925 22:53:54.945886 26566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:54.947073 26566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:54.948266 26566 hybrid_clock.cc:648] HybridClock initialized: now 1790376834948232 us; error 48 us; skew 500 ppm
Sep 25 22:53:54 dist-test-slave-kb1g krb5kdc[26553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:54.954782 26566 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:54.955808 26566 webserver.cc:460] Webserver started at http://127.4.219.190:43355/ using document root <none> and password file <none>
I20260925 22:53:54.956420 26566 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:54.956516 26566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:54.956779 26566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:54.958923 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "bed68de9ea224028b9dd948c07793a46"
format_stamp: "Formatted at 2026-09-25 22:53:54 on dist-test-slave-kb1g"
server_key: "9a4b2198bd1e117272e3c50c1c4cb617"
server_key_iv: "a338402d9023a2807401921fabfb134c"
server_key_version: "encryptionkey@0"
I20260925 22:53:54.959497 26566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "bed68de9ea224028b9dd948c07793a46"
format_stamp: "Formatted at 2026-09-25 22:53:54 on dist-test-slave-kb1g"
server_key: "9a4b2198bd1e117272e3c50c1c4cb617"
server_key_iv: "a338402d9023a2807401921fabfb134c"
server_key_version: "encryptionkey@0"
I20260925 22:53:54.963356 26566 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 22:53:54.965790 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:54.967006 26566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:53:54.967147 26566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "bed68de9ea224028b9dd948c07793a46"
format_stamp: "Formatted at 2026-09-25 22:53:54 on dist-test-slave-kb1g"
server_key: "9a4b2198bd1e117272e3c50c1c4cb617"
server_key_iv: "a338402d9023a2807401921fabfb134c"
server_key_version: "encryptionkey@0"
I20260925 22:53:54.967259 26566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:54.982781 26566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:54.986119 26566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:54.986327 26566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:54.993633 26566 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:36963
I20260925 22:53:54.993626 26633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:36963 every 8 connection(s)
I20260925 22:53:54.994627 26566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:54.997452 26634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:55.001173 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26566
I20260925 22:53:55.001312 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:55.001567 4974 external_mini_cluster.cc:1428] Setting key b0610bb297343b5858c9ef2636669c3d
I20260925 22:53:55.003590 26634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46: Bootstrap starting.
I20260925 22:53:55.006264 26634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:55.007133 26634 log.cc:824] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:55.009295 26634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46: No bootstrap required, opened a new log
Sep 25 22:53:55 dist-test-slave-kb1g krb5kdc[26553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:55.013715 26634 raft_consensus.cc:348] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } }
I20260925 22:53:55.013998 26634 raft_consensus.cc:374] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:55.014062 26634 raft_consensus.cc:729] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bed68de9ea224028b9dd948c07793a46, State: Initialized, Role: FOLLOWER
I20260925 22:53:55.014544 26634 consensus_queue.cc:260] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } }
I20260925 22:53:55.014707 26634 raft_consensus.cc:388] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:55.014781 26634 raft_consensus.cc:482] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:55.014881 26634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:55.016085 26634 raft_consensus.cc:504] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } }
I20260925 22:53:55.016435 26634 leader_election.cc:302] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bed68de9ea224028b9dd948c07793a46; no voters:
I20260925 22:53:55.016801 26634 leader_election.cc:288] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:55.016925 26639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:55.017162 26639 raft_consensus.cc:686] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [term 1 LEADER]: Becoming Leader. State: Replica: bed68de9ea224028b9dd948c07793a46, State: Running, Role: LEADER
I20260925 22:53:55.017499 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:55.002983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54025 (local address 127.4.219.190:36963)
0925 22:53:55.003408 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:55.003431 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:55.003473 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:53:55.004280 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:55.004303 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:55.004755 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:55.005130 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:55.005794 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:55.007299 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:55.008350 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:55.009047 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:55.009201 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:55.012046 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:55.012053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:55.012074 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:55.012115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:55.014205 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:55.014725 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:55.014733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:55.014739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:55.014880 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:55.015269 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:55.015273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:55.015274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:55.015777 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:55.015987 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:55.016235 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 22:53:55.016464 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260925 22:53:55.017508 26639 consensus_queue.cc:237] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } }
I20260925 22:53:55.018087 26634 sys_catalog.cc:564] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:55.019367 26641 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bed68de9ea224028b9dd948c07793a46. Latest consensus state: current_term: 1 leader_uuid: "bed68de9ea224028b9dd948c07793a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } } }
I20260925 22:53:55.019510 26641 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:55.019845 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bed68de9ea224028b9dd948c07793a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bed68de9ea224028b9dd948c07793a46" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36963 } } }
I20260925 22:53:55.019918 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:55.020287 26650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:55.021189 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:55.023535 26650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:55.029304 26650 catalog_manager.cc:1347] Generated new cluster ID: eb17cd91d64e4d72abfb2d65311d7e2f
I20260925 22:53:55.029397 26650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:55.031613 26637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:55.023108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35231 (local address 127.4.219.190:36963)
0925 22:53:55.023219 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:55.023223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:55.023435 (+ 212us) server_negotiation.cc:407] Connection header received
0925 22:53:55.023590 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:55.023593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:55.023645 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:55.023711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:55.024351 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:55.024967 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:55.025899 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:55.026268 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:55.026381 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:55.027985 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:55.027988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:55.027992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:55.028017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:55.029829 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:55.030329 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:55.030333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:55.030335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:55.030418 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:55.030751 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:55.030755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:55.030756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:55.030964 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:55.031120 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:55.031365 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 22:53:55.031460 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:53:55.043356 26650 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:55.044773 26650 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:55.050853 26650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bed68de9ea224028b9dd948c07793a46: Generated new TSK 0
I20260925 22:53:55.051620 26650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:55.059682 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26566
W20260925 22:53:55.065029 26551 connection.cc:535] client connection to 127.4.219.190:36963 recv error: Network error: failed to read from TLS socket (remote: 127.4.219.190:36963): Connection reset by peer (error 104)
2026-09-25T22:53:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:55 dist-test-slave-kb1g krb5kdc[26665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:55 dist-test-slave-kb1g krb5kdc[26665](info): set up 2 sockets
Sep 25 22:53:55 dist-test-slave-kb1g krb5kdc[26665](info): commencing operation
krb5kdc: starting...
W20260925 22:53:56.097141 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:56 dist-test-slave-kb1g krb5kdc[26665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:56.219409 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41063
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:56.335577 26678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:56.335882 26678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:56.335975 26678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:56.340014 26678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:56.340121 26678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:56.340159 26678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:56.340196 26678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:56.340229 26678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:56.345666 26678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:41063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:56.346773 26678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:56.347821 26678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:53:56.356402 26678 server_base.cc:1034] running on GCE node
W20260925 22:53:56.356365 26686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:56.356376 26683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:56.356362 26684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:56.357199 26678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:56.358304 26678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:56.359498 26678 hybrid_clock.cc:648] HybridClock initialized: now 1790376836359482 us; error 34 us; skew 500 ppm
Sep 25 22:53:56 dist-test-slave-kb1g krb5kdc[26665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376836, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:56.362175 26678 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:56.363071 26678 webserver.cc:460] Webserver started at http://127.4.219.190:46371/ using document root <none> and password file <none>
I20260925 22:53:56.363622 26678 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:56.363711 26678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:56.363940 26678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:56.365991 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "2117a96659e34bd0ab486ddaa3956da4"
format_stamp: "Formatted at 2026-09-25 22:53:56 on dist-test-slave-kb1g"
server_key: "5621131a7d6c93740f64934d9b3ebd4b"
server_key_iv: "b574d65150c4312f51be04abf4062872"
server_key_version: "encryptionkey@0"
I20260925 22:53:56.366561 26678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "2117a96659e34bd0ab486ddaa3956da4"
format_stamp: "Formatted at 2026-09-25 22:53:56 on dist-test-slave-kb1g"
server_key: "5621131a7d6c93740f64934d9b3ebd4b"
server_key_iv: "b574d65150c4312f51be04abf4062872"
server_key_version: "encryptionkey@0"
I20260925 22:53:56.370180 26678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:53:56.372476 26693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:56.373731 26678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:56.373868 26678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "2117a96659e34bd0ab486ddaa3956da4"
format_stamp: "Formatted at 2026-09-25 22:53:56 on dist-test-slave-kb1g"
server_key: "5621131a7d6c93740f64934d9b3ebd4b"
server_key_iv: "b574d65150c4312f51be04abf4062872"
server_key_version: "encryptionkey@0"
I20260925 22:53:56.373993 26678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:56.388953 26678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:56.391748 26678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:56.391953 26678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:56.399268 26678 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:41063
I20260925 22:53:56.399319 26745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:41063 every 8 connection(s)
I20260925 22:53:56.400554 26678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:56.403546 26746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:56.406138 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26678
I20260925 22:53:56.406303 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:56.406599 4974 external_mini_cluster.cc:1428] Setting key 7c0b39305746b95e254eb967b1149761
I20260925 22:53:56.409840 26746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4: Bootstrap starting.
I20260925 22:53:56.412032 26746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:56.412866 26746 log.cc:824] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:56.415222 26746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4: No bootstrap required, opened a new log
I20260925 22:53:56.415784 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:56.408029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33519 (local address 127.4.219.190:41063)
0925 22:53:56.408454 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:56.408494 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:56.408574 (+ 80us) server_negotiation.cc:407] Connection header received
0925 22:53:56.409208 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:56.409225 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:56.409574 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:56.409900 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:56.410295 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:56.411236 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:56.412124 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:56.412675 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:56.412835 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:56.413820 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:56.413829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:56.413846 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:56.414330 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:56.414423 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:56.414460 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:56.414610 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:56.414733 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 22:53:56.414866 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":122,"threads_started":1}
I20260925 22:53:56.419297 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:56.419934 26746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } }
I20260925 22:53:56.420231 26746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:56.420298 26746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2117a96659e34bd0ab486ddaa3956da4, State: Initialized, Role: FOLLOWER
I20260925 22:53:56.420801 26746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } }
I20260925 22:53:56.421000 26746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:56.421065 26746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:56.421154 26746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:56.422319 26746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } }
I20260925 22:53:56.422685 26746 leader_election.cc:302] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2117a96659e34bd0ab486ddaa3956da4; no voters:
I20260925 22:53:56.423022 26746 leader_election.cc:288] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:56.423141 26756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:56.423378 26756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [term 1 LEADER]: Becoming Leader. State: Replica: 2117a96659e34bd0ab486ddaa3956da4, State: Running, Role: LEADER
I20260925 22:53:56.423684 26756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } }
I20260925 22:53:56.424304 26746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:56.424852 26756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2117a96659e34bd0ab486ddaa3956da4. Latest consensus state: current_term: 1 leader_uuid: "2117a96659e34bd0ab486ddaa3956da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } } }
I20260925 22:53:56.424973 26756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:56.425482 26757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2117a96659e34bd0ab486ddaa3956da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2117a96659e34bd0ab486ddaa3956da4" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 41063 } } }
I20260925 22:53:56.425561 26757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:56.425808 26760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:56.427361 26749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:56.421777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41087 (local address 127.4.219.190:41063)
0925 22:53:56.421881 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:56.421885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:56.422505 (+ 620us) server_negotiation.cc:407] Connection header received
0925 22:53:56.422645 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:56.422650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:56.422691 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:56.422765 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:56.423114 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:56.423891 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:56.424990 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:56.425377 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:56.425494 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:56.426603 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:56.426607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:56.426610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:56.426622 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:56.426647 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:56.426663 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:56.426754 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:56.427181 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 22:53:56.427216 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:53:56.428915 26760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:56.434543 26760 catalog_manager.cc:1347] Generated new cluster ID: 6a9639245779426fbd1f1300235563a1
I20260925 22:53:56.434615 26760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:56.465456 26760 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:56.466622 26760 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:56.483951 26760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2117a96659e34bd0ab486ddaa3956da4: Generated new TSK 0
I20260925 22:53:56.484815 26760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:56.492839 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26678
W20260925 22:53:56.498639 26663 connection.cc:535] client connection to 127.4.219.190:41063 recv error: Network error: failed to read from TLS socket (remote: 127.4.219.190:41063): Connection reset by peer (error 104)
2026-09-25T22:53:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:56 dist-test-slave-kb1g krb5kdc[26777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:56 dist-test-slave-kb1g krb5kdc[26777](info): set up 2 sockets
Sep 25 22:53:56 dist-test-slave-kb1g krb5kdc[26777](info): commencing operation
krb5kdc: starting...
W20260925 22:53:57.524875 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:57 dist-test-slave-kb1g krb5kdc[26777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:57.647536 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44577
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:57.760368 26790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:57.760742 26790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:57.760818 26790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:57.764828 26790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:57.764883 26790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:57.764904 26790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:57.764948 26790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:57.764984 26790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:57.770303 26790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44577
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:57.771333 26790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:57.772344 26790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:57.780714 26795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:57.780758 26790 server_base.cc:1034] running on GCE node
W20260925 22:53:57.780722 26796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:57.780731 26798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:57.781535 26790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:57.782588 26790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:57.783771 26790 hybrid_clock.cc:648] HybridClock initialized: now 1790376837783751 us; error 30 us; skew 500 ppm
Sep 25 22:53:57 dist-test-slave-kb1g krb5kdc[26777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:57.786509 26790 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:57.787410 26790 webserver.cc:460] Webserver started at http://127.4.219.190:36225/ using document root <none> and password file <none>
I20260925 22:53:57.787961 26790 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:57.788059 26790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:57.788286 26790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:57.790359 26790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "099440f22ffc4472afb38390f02e927d"
format_stamp: "Formatted at 2026-09-25 22:53:57 on dist-test-slave-kb1g"
server_key: "ec14b35b504e35dfc714c5022d6d165c"
server_key_iv: "28b29fae8ee285ac23a7267eda7f7974"
server_key_version: "encryptionkey@0"
I20260925 22:53:57.790908 26790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "099440f22ffc4472afb38390f02e927d"
format_stamp: "Formatted at 2026-09-25 22:53:57 on dist-test-slave-kb1g"
server_key: "ec14b35b504e35dfc714c5022d6d165c"
server_key_iv: "28b29fae8ee285ac23a7267eda7f7974"
server_key_version: "encryptionkey@0"
I20260925 22:53:57.794624 26790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:53:57.797010 26805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:57.798244 26790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:53:57.798395 26790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "099440f22ffc4472afb38390f02e927d"
format_stamp: "Formatted at 2026-09-25 22:53:57 on dist-test-slave-kb1g"
server_key: "ec14b35b504e35dfc714c5022d6d165c"
server_key_iv: "28b29fae8ee285ac23a7267eda7f7974"
server_key_version: "encryptionkey@0"
I20260925 22:53:57.798508 26790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:57.803089 26790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:57.806046 26790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:57.806239 26790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:57.813149 26790 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44577
I20260925 22:53:57.813169 26857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44577 every 8 connection(s)
I20260925 22:53:57.814069 26790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:57.814615 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26790
I20260925 22:53:57.814801 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:57.815085 4974 external_mini_cluster.cc:1428] Setting key c63e99717a641ff5ed3eef2807473c76
I20260925 22:53:57.817219 26858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:57.820633 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:57.816464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51741 (local address 127.4.219.190:44577)
0925 22:53:57.816917 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:57.816937 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:57.816979 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:53:57.817793 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:57.817816 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:57.818167 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:57.818535 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:57.819258 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:57.819263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:57.819281 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:57.819292 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:57.819351 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:57.819375 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:57.819505 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:57.819595 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:53:57.819738 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":120,"threads_started":1}
I20260925 22:53:57.823398 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:57.823930 26858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: Bootstrap starting.
I20260925 22:53:57.826372 26858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:57.827101 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:57.825551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40871 (local address 127.4.219.190:44577)
0925 22:53:57.825668 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:57.825672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:57.825798 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:53:57.825892 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:57.825895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:57.825928 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:57.826009 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:57.826722 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:57.826726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:57.826729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:53:57.826740 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:57.826767 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:53:57.826779 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:57.826867 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:57.826931 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:53:57.826966 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:53:57.827234 26858 log.cc:824] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:57.829260 26858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: No bootstrap required, opened a new log
I20260925 22:53:57.832790 26858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } }
I20260925 22:53:57.833045 26858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:57.833127 26858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099440f22ffc4472afb38390f02e927d, State: Initialized, Role: FOLLOWER
I20260925 22:53:57.833647 26858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } }
I20260925 22:53:57.833815 26858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:57.833886 26858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:57.833992 26858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:57.834954 26858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } }
I20260925 22:53:57.835253 26858 leader_election.cc:302] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 099440f22ffc4472afb38390f02e927d; no voters:
I20260925 22:53:57.835582 26858 leader_election.cc:288] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:57.835798 26868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:57.836123 26868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [term 1 LEADER]: Becoming Leader. State: Replica: 099440f22ffc4472afb38390f02e927d, State: Running, Role: LEADER
I20260925 22:53:57.836478 26868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } }
I20260925 22:53:57.837037 26858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:57.838588 26869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "099440f22ffc4472afb38390f02e927d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } } }
I20260925 22:53:57.838752 26869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:57.838588 26870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 099440f22ffc4472afb38390f02e927d. Latest consensus state: current_term: 1 leader_uuid: "099440f22ffc4472afb38390f02e927d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "099440f22ffc4472afb38390f02e927d" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44577 } } }
I20260925 22:53:57.838837 26870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:57.842180 26883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:57.842279 26883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:57.842365 26881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:57.843072 26881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:57.848877 26881 catalog_manager.cc:1347] Generated new cluster ID: 2f9d99b1a53a409ab51585d7ce9be653
I20260925 22:53:57.848958 26881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:57.860131 26881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:57.860495 26881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:57.865965 26881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 099440f22ffc4472afb38390f02e927d: Generated new TSK 0
I20260925 22:53:57.866736 26881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:57.872875 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26790
2026-09-25T22:53:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:57 dist-test-slave-kb1g krb5kdc[26889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:57 dist-test-slave-kb1g krb5kdc[26889](info): set up 2 sockets
Sep 25 22:53:57 dist-test-slave-kb1g krb5kdc[26889](info): commencing operation
krb5kdc: starting...
W20260925 22:53:58.916877 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[26889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:53:59.047905 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39931
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:53:59.156975 26902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:53:59.157270 26902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:53:59.157351 26902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:53:59.161366 26902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:53:59.161463 26902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:53:59.161502 26902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:53:59.161558 26902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:53:59.161594 26902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:53:59.166889 26902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:39931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:53:59.168016 26902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:53:59.169085 26902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:53:59.177405 26908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:59.177405 26910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:53:59.177402 26907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:53:59.177745 26902 server_base.cc:1034] running on GCE node
I20260925 22:53:59.178277 26902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:53:59.179342 26902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:53:59.180516 26902 hybrid_clock.cc:648] HybridClock initialized: now 1790376839180502 us; error 25 us; skew 500 ppm
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[26889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376839, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:53:59.183533 26902 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:53:59.184423 26902 webserver.cc:460] Webserver started at http://127.4.219.190:43753/ using document root <none> and password file <none>
I20260925 22:53:59.184999 26902 fs_manager.cc:359] Metadata directory not provided
I20260925 22:53:59.185061 26902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:53:59.185302 26902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:53:59.187378 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "cd7e1ea3720343b3a3abb4b4d3cec499"
format_stamp: "Formatted at 2026-09-25 22:53:59 on dist-test-slave-kb1g"
server_key: "5beb18becda3968b86b785d3a362c683"
server_key_iv: "ddac2a03a72c21427789c6f4f6ff9c3a"
server_key_version: "encryptionkey@0"
I20260925 22:53:59.187956 26902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "cd7e1ea3720343b3a3abb4b4d3cec499"
format_stamp: "Formatted at 2026-09-25 22:53:59 on dist-test-slave-kb1g"
server_key: "5beb18becda3968b86b785d3a362c683"
server_key_iv: "ddac2a03a72c21427789c6f4f6ff9c3a"
server_key_version: "encryptionkey@0"
I20260925 22:53:59.191921 26902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:53:59.194578 26917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:53:59.195933 26902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:53:59.196069 26902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "cd7e1ea3720343b3a3abb4b4d3cec499"
format_stamp: "Formatted at 2026-09-25 22:53:59 on dist-test-slave-kb1g"
server_key: "5beb18becda3968b86b785d3a362c683"
server_key_iv: "ddac2a03a72c21427789c6f4f6ff9c3a"
server_key_version: "encryptionkey@0"
I20260925 22:53:59.196182 26902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:53:59.211613 26902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:53:59.214988 26902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:53:59.215197 26902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:53:59.222548 26902 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:39931
I20260925 22:53:59.222574 26969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:39931 every 8 connection(s)
I20260925 22:53:59.223495 26902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:53:59.224116 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 26902
I20260925 22:53:59.224262 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:53:59.224566 4974 external_mini_cluster.cc:1428] Setting key 71c13294e789bca1ac9daff98948eca9
I20260925 22:53:59.226792 26970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:53:59.232167 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: Bootstrap starting.
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[26889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:53:59.234527 26970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: Neither blocks nor log segments found. Creating new log.
I20260925 22:53:59.235368 26970 log.cc:824] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: Log is configured to *not* fsync() on all Append() calls
I20260925 22:53:59.237607 26970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: No bootstrap required, opened a new log
I20260925 22:53:59.238487 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:59.226027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53125 (local address 127.4.219.190:39931)
0925 22:53:59.226475 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:59.226492 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:59.226528 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:53:59.227203 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:59.227221 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:59.227565 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:59.227911 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:59.228763 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:59.229745 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:59.230627 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:59.231096 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:59.231265 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:59.233860 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:59.233865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:59.233877 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:59.233903 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:59.235777 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:59.236244 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:59.236249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:59.236253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:59.236357 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:59.236643 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:59.236645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:59.236646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:59.237020 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:59.237166 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:59.237466 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 22:53:59.237778 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":141,"threads_started":1}
I20260925 22:53:59.241492 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:53:59.241688 26970 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } }
I20260925 22:53:59.241946 26970 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:53:59.242008 26970 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7e1ea3720343b3a3abb4b4d3cec499, State: Initialized, Role: FOLLOWER
I20260925 22:53:59.242483 26970 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } }
I20260925 22:53:59.242662 26970 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:53:59.242734 26970 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:53:59.242849 26970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:53:59.244072 26970 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } }
I20260925 22:53:59.244442 26970 leader_election.cc:302] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7e1ea3720343b3a3abb4b4d3cec499; no voters:
I20260925 22:53:59.244776 26970 leader_election.cc:288] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:53:59.244890 26980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:53:59.245123 26980 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [term 1 LEADER]: Becoming Leader. State: Replica: cd7e1ea3720343b3a3abb4b4d3cec499, State: Running, Role: LEADER
I20260925 22:53:59.245512 26980 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } }
I20260925 22:53:59.246133 26970 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:53:59.246688 26980 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7e1ea3720343b3a3abb4b4d3cec499. Latest consensus state: current_term: 1 leader_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } } }
I20260925 22:53:59.247396 26980 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [sys.catalog]: This master's current role is: LEADER
I20260925 22:53:59.247270 26982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7e1ea3720343b3a3abb4b4d3cec499" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 39931 } } }
I20260925 22:53:59.247704 26982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499 [sys.catalog]: This master's current role is: LEADER
W20260925 22:53:59.252084 26995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:53:59.252203 26995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:53:59.252337 26989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:53:59.252370 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:53:59.243799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40099 (local address 127.4.219.190:39931)
0925 22:53:59.243938 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:53:59.243944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:53:59.244076 (+ 132us) server_negotiation.cc:407] Connection header received
0925 22:53:59.244150 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:53:59.244156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:53:59.244203 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:53:59.244279 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:53:59.245089 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:59.245839 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:59.246992 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:53:59.247399 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:53:59.247500 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:53:59.249206 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:53:59.249209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:53:59.249212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:53:59.249231 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:53:59.250561 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:59.251007 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:59.251011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:59.251013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:59.251105 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:53:59.251387 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:53:59.251392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:53:59.251393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:53:59.251661 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:53:59.251788 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:53:59.252048 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 22:53:59.252169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:53:59.253206 26989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:53:59.258666 26989 catalog_manager.cc:1347] Generated new cluster ID: f03e66b4824e42ed98e72151360df64e
I20260925 22:53:59.258733 26989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:53:59.271019 26989 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:53:59.272292 26989 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:53:59.283483 26989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7e1ea3720343b3a3abb4b4d3cec499: Generated new TSK 0
I20260925 22:53:59.284327 26989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:53:59.292032 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 26902
2026-09-25T22:53:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[27001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[27001](info): set up 2 sockets
Sep 25 22:53:59 dist-test-slave-kb1g krb5kdc[27001](info): commencing operation
krb5kdc: starting...
W20260925 22:54:00.340967 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:54:00 dist-test-slave-kb1g krb5kdc[27001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:54:00.468458 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35437
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:35437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:00.578981 27014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:00.579247 27014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:00.579290 27014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:00.583271 27014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:00.583329 27014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:00.583349 27014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:00.583366 27014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:00.583431 27014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:00.588645 27014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:35437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:00.589745 27014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:00.590760 27014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:00.599139 27019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:00.599160 27020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:00.599160 27022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:00.599339 27014 server_base.cc:1034] running on GCE node
I20260925 22:54:00.600023 27014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:00.601190 27014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:00.602396 27014 hybrid_clock.cc:648] HybridClock initialized: now 1790376840602364 us; error 52 us; skew 500 ppm
Sep 25 22:54:00 dist-test-slave-kb1g krb5kdc[27001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:00.608485 27014 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:54:00.609463 27014 webserver.cc:460] Webserver started at http://127.4.219.190:43757/ using document root <none> and password file <none>
I20260925 22:54:00.610051 27014 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:00.610146 27014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:00.610378 27014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:00.612444 27014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "fc849c29814a4163a4a6faead190dfdc"
format_stamp: "Formatted at 2026-09-25 22:54:00 on dist-test-slave-kb1g"
server_key: "8625d9c8ecebc627f9f3e99db8d452fc"
server_key_iv: "3f06672e901e707424c71d85e5087c6c"
server_key_version: "encryptionkey@0"
I20260925 22:54:00.613081 27014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "fc849c29814a4163a4a6faead190dfdc"
format_stamp: "Formatted at 2026-09-25 22:54:00 on dist-test-slave-kb1g"
server_key: "8625d9c8ecebc627f9f3e99db8d452fc"
server_key_iv: "3f06672e901e707424c71d85e5087c6c"
server_key_version: "encryptionkey@0"
I20260925 22:54:00.616895 27014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:54:00.619264 27029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:00.620483 27014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:00.620639 27014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "fc849c29814a4163a4a6faead190dfdc"
format_stamp: "Formatted at 2026-09-25 22:54:00 on dist-test-slave-kb1g"
server_key: "8625d9c8ecebc627f9f3e99db8d452fc"
server_key_iv: "3f06672e901e707424c71d85e5087c6c"
server_key_version: "encryptionkey@0"
I20260925 22:54:00.620752 27014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:00.631951 27014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:00.634763 27014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:00.634967 27014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:00.641532 27014 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:35437
I20260925 22:54:00.641562 27081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:35437 every 8 connection(s)
I20260925 22:54:00.642509 27014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:00.644824 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27014
I20260925 22:54:00.644969 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:54:00.645251 4974 external_mini_cluster.cc:1428] Setting key ac0ff3e2c6c1ec0dd3d9c3b792fe78d6
I20260925 22:54:00.645356 27082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:00.652390 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc: Bootstrap starting.
I20260925 22:54:00.655058 27082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:00.655319 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:00.646620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48865 (local address 127.4.219.190:35437)
0925 22:54:00.647020 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:00.647037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:00.647078 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:54:00.648014 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:00.648038 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:00.648471 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:00.648864 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:00.649267 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:00.650725 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:00.651638 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:00.652324 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:00.652501 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:00.653346 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:00.653353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:00.653367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:00.653877 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:00.653941 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:00.653969 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:00.654063 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:00.654195 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:54:00.654389 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260925 22:54:00.655930 27082 log.cc:824] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:00.658107 27082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc: No bootstrap required, opened a new log
I20260925 22:54:00.658282 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:00.662662 27082 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } }
I20260925 22:54:00.662946 27082 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:00.663012 27082 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc849c29814a4163a4a6faead190dfdc, State: Initialized, Role: FOLLOWER
I20260925 22:54:00.663513 27082 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } }
I20260925 22:54:00.663684 27082 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:00.663758 27082 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:00.663864 27082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:00.665088 27085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:00.660555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50429 (local address 127.4.219.190:35437)
0925 22:54:00.660691 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:00.660695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:00.660987 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:54:00.661086 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:00.661088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:00.661126 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:00.661196 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:00.661535 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:00.662027 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:00.663097 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:00.663435 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:00.663509 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:00.664644 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:00.664647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:00.664649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:00.664661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:00.664684 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:00.664694 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:00.664758 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:00.664892 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:54:00.664937 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:54:00.665128 27082 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } }
I20260925 22:54:00.665524 27082 leader_election.cc:302] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc849c29814a4163a4a6faead190dfdc; no voters:
I20260925 22:54:00.665848 27082 leader_election.cc:288] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:00.665958 27092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:00.666180 27092 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [term 1 LEADER]: Becoming Leader. State: Replica: fc849c29814a4163a4a6faead190dfdc, State: Running, Role: LEADER
I20260925 22:54:00.666493 27092 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } }
I20260925 22:54:00.667107 27082 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:00.668040 27094 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc849c29814a4163a4a6faead190dfdc. Latest consensus state: current_term: 1 leader_uuid: "fc849c29814a4163a4a6faead190dfdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } } }
I20260925 22:54:00.668319 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc849c29814a4163a4a6faead190dfdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc849c29814a4163a4a6faead190dfdc" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 35437 } } }
I20260925 22:54:00.668440 27094 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:00.668510 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:00.668963 27101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:00.671537 27101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:00.677013 27101 catalog_manager.cc:1347] Generated new cluster ID: 0f694b80c38f4b098f9618154b9cf433
I20260925 22:54:00.677104 27101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:00.696925 27101 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:00.697832 27101 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:00.702009 27101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc849c29814a4163a4a6faead190dfdc: Generated new TSK 0
I20260925 22:54:00.702688 27101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:00.718087 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27014
2026-09-25T22:54:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:00 dist-test-slave-kb1g krb5kdc[27113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:00 dist-test-slave-kb1g krb5kdc[27113](info): set up 2 sockets
Sep 25 22:54:00 dist-test-slave-kb1g krb5kdc[27113](info): commencing operation
krb5kdc: starting...
W20260925 22:54:01.748827 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:54:01 dist-test-slave-kb1g krb5kdc[27113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:54:01.871428 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36253
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:36253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:01.977631 27126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:01.977865 27126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:01.977906 27126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:01.981737 27126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:01.981786 27126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:01.981804 27126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:01.981820 27126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:01.981835 27126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:01.986783 27126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:36253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:36253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:01.987718 27126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:01.988660 27126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:01.996833 27132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:01.996966 27126 server_base.cc:1034] running on GCE node
W20260925 22:54:01.996866 27134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:01.996896 27131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:01.997691 27126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:01.998817 27126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:01.999985 27126 hybrid_clock.cc:648] HybridClock initialized: now 1790376841999953 us; error 43 us; skew 500 ppm
Sep 25 22:54:01 dist-test-slave-kb1g krb5kdc[27113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376841, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:02.002714 27126 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:54:02.003616 27126 webserver.cc:460] Webserver started at http://127.4.219.190:34235/ using document root <none> and password file <none>
I20260925 22:54:02.004154 27126 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:02.004212 27126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:02.004446 27126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:02.006489 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "9f88126c6b3645adac27568871d4aafa"
format_stamp: "Formatted at 2026-09-25 22:54:02 on dist-test-slave-kb1g"
server_key: "94b89fb9b5ba3f5e39902b047db7fb48"
server_key_iv: "cb3ddfb0c051782e88b045cd25decbac"
server_key_version: "encryptionkey@0"
I20260925 22:54:02.007043 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "9f88126c6b3645adac27568871d4aafa"
format_stamp: "Formatted at 2026-09-25 22:54:02 on dist-test-slave-kb1g"
server_key: "94b89fb9b5ba3f5e39902b047db7fb48"
server_key_iv: "cb3ddfb0c051782e88b045cd25decbac"
server_key_version: "encryptionkey@0"
I20260925 22:54:02.010648 27126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260925 22:54:02.013002 27141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:02.014204 27126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:02.014341 27126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "9f88126c6b3645adac27568871d4aafa"
format_stamp: "Formatted at 2026-09-25 22:54:02 on dist-test-slave-kb1g"
server_key: "94b89fb9b5ba3f5e39902b047db7fb48"
server_key_iv: "cb3ddfb0c051782e88b045cd25decbac"
server_key_version: "encryptionkey@0"
I20260925 22:54:02.014478 27126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:02.019035 27126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:02.021852 27126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:02.022033 27126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:02.028817 27126 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:36253
I20260925 22:54:02.028841 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:36253 every 8 connection(s)
I20260925 22:54:02.029695 27126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:02.032240 27194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:02.037063 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27126
I20260925 22:54:02.037221 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:54:02.037503 4974 external_mini_cluster.cc:1428] Setting key be92b5939f90157413ba012e579dd162
I20260925 22:54:02.037788 27194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: Bootstrap starting.
I20260925 22:54:02.040205 27194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:02.041013 27194 log.cc:824] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:02.042632 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:02.038900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42823 (local address 127.4.219.190:36253)
0925 22:54:02.039355 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:02.039370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:02.039406 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:54:02.040028 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:02.040044 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:02.040363 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:02.040736 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:02.041488 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:02.041493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:02.041508 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:02.041519 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:02.041577 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:02.041597 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:02.041716 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:02.041826 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:54:02.041979 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":133,"threads_started":1}
I20260925 22:54:02.042918 27194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: No bootstrap required, opened a new log
I20260925 22:54:02.045483 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:02.047001 27194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } }
I20260925 22:54:02.047232 27194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:54:02.047294 27194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88126c6b3645adac27568871d4aafa, State: Initialized, Role: FOLLOWER
I20260925 22:54:02.047747 27194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } }
I20260925 22:54:02.047917 27194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:54:02.048003 27194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:54:02.048102 27194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:02.049093 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:02.047429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51781 (local address 127.4.219.190:36253)
0925 22:54:02.047558 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:02.047562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:02.047702 (+ 140us) server_negotiation.cc:407] Connection header received
0925 22:54:02.047779 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:02.047782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:02.047815 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:02.047898 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:02.048731 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:02.048734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:02.048737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:54:02.048748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:02.048774 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:54:02.048786 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:02.048873 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:02.048933 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 22:54:02.048970 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:02.049302 27194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } }
I20260925 22:54:02.049626 27194 leader_election.cc:302] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f88126c6b3645adac27568871d4aafa; no voters:
I20260925 22:54:02.049932 27194 leader_election.cc:288] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:54:02.049993 27204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:02.050235 27204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [term 1 LEADER]: Becoming Leader. State: Replica: 9f88126c6b3645adac27568871d4aafa, State: Running, Role: LEADER
I20260925 22:54:02.050558 27204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } }
I20260925 22:54:02.051208 27194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:02.052153 27205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f88126c6b3645adac27568871d4aafa. Latest consensus state: current_term: 1 leader_uuid: "9f88126c6b3645adac27568871d4aafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } } }
I20260925 22:54:02.052277 27205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:02.052613 27206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f88126c6b3645adac27568871d4aafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f88126c6b3645adac27568871d4aafa" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 36253 } } }
I20260925 22:54:02.052703 27206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa [sys.catalog]: This master's current role is: LEADER
W20260925 22:54:02.055804 27219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:02.055907 27219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:02.056033 27213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:02.056912 27213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:02.062165 27213 catalog_manager.cc:1347] Generated new cluster ID: 69f62f933faf4bf184ef18b973606f59
I20260925 22:54:02.062224 27213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:02.072019 27213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:02.072310 27213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:02.076601 27213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f88126c6b3645adac27568871d4aafa: Generated new TSK 0
I20260925 22:54:02.077227 27213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:54:02.082221 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27126
2026-09-25T22:54:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1363 ms)
[----------] 18 tests from AuthTokenIssuingTest (25399 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:02 dist-test-slave-kb1g krb5kdc[27225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:02 dist-test-slave-kb1g krb5kdc[27225](info): set up 2 sockets
Sep 25 22:54:02 dist-test-slave-kb1g krb5kdc[27225](info): commencing operation
krb5kdc: starting...
W20260925 22:54:03.120867 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:54:03.239785 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:33077
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:03.349303 27238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:03.349592 27238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:03.349617 27238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:03.349701 27238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:03.353722 27238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:03.353777 27238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:03.353796 27238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:03.353813 27238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:03.353865 27238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:03.359112 27238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:03.360172 27238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:03.361291 27238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:03.370323 27243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:03.370373 27244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:03.370326 27246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:03.370616 27238 server_base.cc:1034] running on GCE node
I20260925 22:54:03.371186 27238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:03.372259 27238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:03.373411 27238 hybrid_clock.cc:648] HybridClock initialized: now 1790376843373382 us; error 44 us; skew 500 ppm
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:03.376101 27238 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:54:03.377075 27238 webserver.cc:460] Webserver started at http://127.4.219.190:40503/ using document root <none> and password file <none>
I20260925 22:54:03.377626 27238 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:03.377717 27238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:03.377948 27238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:03.379961 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "1af20b4b3a924243828e20a910ce904a"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "bab6d5ee173ed38e7073687bb454ac19"
server_key_iv: "45833d15eaab4d9437db38c617953787"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.380519 27238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "1af20b4b3a924243828e20a910ce904a"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "bab6d5ee173ed38e7073687bb454ac19"
server_key_iv: "45833d15eaab4d9437db38c617953787"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.384155 27238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:03.386685 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:03.387936 27238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 22:54:03.388084 27238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "1af20b4b3a924243828e20a910ce904a"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "bab6d5ee173ed38e7073687bb454ac19"
server_key_iv: "45833d15eaab4d9437db38c617953787"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.388198 27238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:03.418905 27238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:03.421826 27238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:03.422027 27238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:03.429747 27238 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:33077
I20260925 22:54:03.430022 27305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:33077 every 8 connection(s)
I20260925 22:54:03.431227 27238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:03.435061 27306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:03.436067 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27238
I20260925 22:54:03.436228 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:54:03.436513 4974 external_mini_cluster.cc:1428] Setting key 909cffc43d14f9a45a5942519e7e8633
I20260925 22:54:03.440440 27306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.219.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.189@KRBTEST.COM" created.
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:54:03.456635 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.443185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.4.219.190:33077)
0925 22:54:03.443752 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.443763 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.443808 (+ 45us) server_negotiation.cc:407] Connection header received
0925 22:54:03.445015 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.445025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.445274 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.445499 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.446317 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.447334 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.448422 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.448909 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.449074 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.451706 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.451711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.451721 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.451741 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.453428 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.454055 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.454060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.454061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.454167 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.454545 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.454547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.454549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.454932 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.455037 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.455500 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.455771 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":449,"thread_start_us":154,"threads_started":1}
I20260925 22:54:03.456641 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.443184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33077 (local address 127.0.0.1:40430)
0925 22:54:03.443651 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.443701 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.444681 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.445518 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.445523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.445858 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.446115 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.446128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.447459 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.447462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.448304 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.448313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.449056 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.449060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.449169 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.449779 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.449795 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.451558 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.453543 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.453555 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.453568 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.453923 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.454270 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.454276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.454278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.454436 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.455057 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.455064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.455234 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.455435 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.455877 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":100,"thread_start_us":129,"threads_started":1}
I20260925 22:54:03.459136 27306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:03.459919 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.459668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47734)
0925 22:54:03.459783 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.459863 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 22:54:03.460127 27256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
I20260925 22:54:03.461115 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.460892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47738)
0925 22:54:03.460984 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.461026 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 22:54:03.461295 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:46615: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
I20260925 22:54:03.467881 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:46615
--webserver_interface=127.4.219.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:03.510578 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } attempt: 1
I20260925 22:54:03.511488 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.511183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47750)
0925 22:54:03.511340 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.511409 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:54:03.512238 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.512022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47758)
0925 22:54:03.512131 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.512182 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:54:03.512367 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:46615: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
I20260925 22:54:03.580593 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } attempt: 2
I20260925 22:54:03.581396 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.581106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47760)
0925 22:54:03.581250 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.581318 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:54:03.581231 27313 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:03.581538 27313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:03.581634 27313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:03.581696 27313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:54:03.582127 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.581917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47764)
0925 22:54:03.582023 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.582073 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:54:03.582279 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:46615: Network error: Client connection negotiation failed: client connection to 127.4.219.189:46615: connect: Connection refused (error 111)
W20260925 22:54:03.585981 27313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:03.586045 27313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:03.586066 27313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:03.586082 27313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:03.586136 27313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:03.592423 27313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.219.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:03.593653 27313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:03.594664 27313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:03.602954 27319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:03.603148 27313 server_base.cc:1034] running on GCE node
W20260925 22:54:03.603001 27318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:03.602954 27321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:03.603926 27313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:03.605060 27313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:03.606256 27313 hybrid_clock.cc:648] HybridClock initialized: now 1790376843606233 us; error 38 us; skew 500 ppm
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:03.609055 27313 init.cc:377] Logged in from keytab as kudu/127.4.219.189@KRBTEST.COM (short username kudu)
I20260925 22:54:03.609925 27313 webserver.cc:460] Webserver started at http://127.4.219.189:34129/ using document root <none> and password file <none>
I20260925 22:54:03.610491 27313 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:03.610580 27313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:03.610827 27313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:03.612923 27313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data/instance:
uuid: "e33ee51f5cc647f5b7eed3ff7a465219"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "f644bfca0214120fae53fd674c8b85f0"
server_key_iv: "64684f0835b3dde4586fbc8fb18ce346"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.613498 27313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal/instance:
uuid: "e33ee51f5cc647f5b7eed3ff7a465219"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "f644bfca0214120fae53fd674c8b85f0"
server_key_iv: "64684f0835b3dde4586fbc8fb18ce346"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.617355 27313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:54:03.619979 27328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:03.621424 27313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:54:03.621574 27313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "e33ee51f5cc647f5b7eed3ff7a465219"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "f644bfca0214120fae53fd674c8b85f0"
server_key_iv: "64684f0835b3dde4586fbc8fb18ce346"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.621695 27313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:03.649310 27313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:03.652397 27313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:03.652631 27313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:03.659659 27313 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.189:46615
I20260925 22:54:03.659955 27380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.189:46615 every 8 connection(s)
I20260925 22:54:03.660688 27313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
I20260925 22:54:03.663569 27381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:03.664480 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27313
I20260925 22:54:03.664640 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-1/wal/instance
I20260925 22:54:03.664927 4974 external_mini_cluster.cc:1428] Setting key dc6e95e0283e38258479d74d66a1afda
I20260925 22:54:03.667987 27381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.219.188@KRBTEST.COM; defaulting to no policy
I20260925 22:54:03.673437 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.188@KRBTEST.COM" created.
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:03.684115 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.670386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.4.219.190:33077)
0925 22:54:03.670545 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.670550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.671387 (+ 837us) server_negotiation.cc:407] Connection header received
0925 22:54:03.672290 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.672294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.672372 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.672482 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.673820 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.674453 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.676000 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.676264 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.676371 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.679297 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.679303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.679310 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.679331 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.681800 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682484 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.682490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.682492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.682562 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682982 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.682987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.682989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.683194 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.683295 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.683816 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.683938 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:54:03.684136 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.673847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47766)
0925 22:54:03.673962 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.673976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.674043 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.674765 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.674768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.674961 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.675067 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.675074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.676407 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.676414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.677313 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.677322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.677881 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.677885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.677999 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.678540 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.678555 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.680104 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.682397 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.682404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.682657 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.682935 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.682942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.683030 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.683718 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.683723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.683815 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.683915 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.684022 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:54:03.685213 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.670764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33077 (local address 127.0.0.1:40442)
0925 22:54:03.671216 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.671266 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.672089 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.672694 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.672701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.673068 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.673666 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.673682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.674574 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.674580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.675882 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.675892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.676420 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.676424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.676656 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.677291 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.677315 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.679164 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.681919 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.681928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.681937 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.682381 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.682663 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.682669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.682849 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.683293 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.683300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.683569 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.684278 (+ 709us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.684471 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":3,"thread_start_us":120,"threads_started":1}
I20260925 22:54:03.685213 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.674036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.4.219.189:46615)
0925 22:54:03.674343 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.674352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.674371 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:54:03.674403 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.674407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.674550 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.674742 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.675208 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.676270 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.677430 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.677759 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.677870 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.680226 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.680232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.680240 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.680260 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.682295 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.682738 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.682743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.682745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.682839 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.683134 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.683138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.683142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.683600 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.683738 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.684035 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.684367 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":6400,"thread_start_us":146,"threads_started":1}
I20260925 22:54:03.686667 27381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:03.687124 27306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:03.687744 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.687523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60736)
0925 22:54:03.687631 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.687678 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 22:54:03.688453 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.688178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60752)
0925 22:54:03.688327 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.688389 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 22:54:03.688637 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
I20260925 22:54:03.692917 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:45571
--webserver_interface=127.4.219.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:03.695398 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.687330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.4.219.189:46615)
0925 22:54:03.687488 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.687492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.687502 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:54:03.687531 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.687538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.687568 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.687649 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.688058 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.688766 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.689609 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.689896 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.690008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.692062 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.692065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.692067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.692086 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.693726 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.694109 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.694114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.694116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.694209 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.694567 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.694569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.694570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.694722 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.694796 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.695086 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.695257 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":56}
I20260925 22:54:03.695545 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.687172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47770)
0925 22:54:03.687287 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.687300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.687396 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.687659 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.687662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.687833 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.687948 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.687955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.688884 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.688889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.689496 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.689504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.690003 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.690007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.690100 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.690586 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.690599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.691938 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.693835 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.693838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.693839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.694016 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.694314 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.694319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.694323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.694477 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.694812 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.694816 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.694924 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.695043 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.695190 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 22:54:03.696283 27381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:03.696980 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.696732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60762)
0925 22:54:03.696851 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.696921 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:54:03.697707 27329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
I20260925 22:54:03.698645 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.698461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60768)
0925 22:54:03.698561 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.698601 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:54:03.698885 27381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
I20260925 22:54:03.719825 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 1
I20260925 22:54:03.720813 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.720455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60770)
0925 22:54:03.720645 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.720715 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:54:03.721570 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.721353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60784)
0925 22:54:03.721460 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.721511 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 22:54:03.721706 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
I20260925 22:54:03.748185 27381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 1
I20260925 22:54:03.749125 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.748830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60796)
0925 22:54:03.748980 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.749047 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:54:03.749842 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.749629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60798)
0925 22:54:03.749736 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.749791 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:54:03.749969 27381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
I20260925 22:54:03.796905 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 2
I20260925 22:54:03.797808 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.797485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60806)
0925 22:54:03.797649 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.797714 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:54:03.798553 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.798337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60820)
0925 22:54:03.798448 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.798500 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:54:03.798686 27306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
W20260925 22:54:03.804131 27387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:03.804445 27387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:03.804507 27387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:03.804612 27387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:03.808651 27387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:03.808722 27387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:03.808765 27387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:03.808800 27387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:03.808837 27387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:03.814049 27387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:33077,127.4.219.189:46615,127.4.219.188:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.219.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:03.815090 27387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:03.816105 27387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:03.818130 27381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 2
I20260925 22:54:03.818955 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.818644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60822)
0925 22:54:03.818809 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.818877 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:54:03.819669 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.819450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60832)
0925 22:54:03.819562 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.819613 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 22:54:03.819800 27381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:45571: Network error: Client connection negotiation failed: client connection to 127.4.219.188:45571: connect: Connection refused (error 111)
W20260925 22:54:03.824420 27393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:03.824483 27387 server_base.cc:1034] running on GCE node
W20260925 22:54:03.824429 27394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:03.824429 27396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:03.825227 27387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:03.826399 27387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:03.827597 27387 hybrid_clock.cc:648] HybridClock initialized: now 1790376843827564 us; error 42 us; skew 500 ppm
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:03.830493 27387 init.cc:377] Logged in from keytab as kudu/127.4.219.188@KRBTEST.COM (short username kudu)
I20260925 22:54:03.831388 27387 webserver.cc:460] Webserver started at http://127.4.219.188:33637/ using document root <none> and password file <none>
I20260925 22:54:03.831952 27387 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:03.832042 27387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:03.832267 27387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:03.834368 27387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data/instance:
uuid: "2dc4d452d0384346bdbabaff61316dbd"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "c6d99096a3d0fa6da8fb721edbc5e5aa"
server_key_iv: "ea65ba6e727ac961a86a7789752f6ea2"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.834931 27387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal/instance:
uuid: "2dc4d452d0384346bdbabaff61316dbd"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "c6d99096a3d0fa6da8fb721edbc5e5aa"
server_key_iv: "ea65ba6e727ac961a86a7789752f6ea2"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.838680 27387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:54:03.841094 27403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:03.842329 27387 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:03.842479 27387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "2dc4d452d0384346bdbabaff61316dbd"
format_stamp: "Formatted at 2026-09-25 22:54:03 on dist-test-slave-kb1g"
server_key: "c6d99096a3d0fa6da8fb721edbc5e5aa"
server_key_iv: "ea65ba6e727ac961a86a7789752f6ea2"
server_key_version: "encryptionkey@0"
I20260925 22:54:03.842593 27387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:03.858987 27387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:03.862076 27387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:03.862277 27387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:03.869202 27387 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.188:45571
I20260925 22:54:03.869520 27455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.188:45571 every 8 connection(s)
I20260925 22:54:03.870170 27387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
I20260925 22:54:03.871181 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27387
I20260925 22:54:03.871325 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376566921023-4974-0/minicluster-data/master-2/wal/instance
I20260925 22:54:03.871593 4974 external_mini_cluster.cc:1428] Setting key ecf3babc89fad04782d15834f1efcf80
I20260925 22:54:03.873359 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:03.877539 27456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:54:03.886555 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.872976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.4.219.188:45571)
0925 22:54:03.873362 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.873386 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.873422 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:54:03.874108 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.874125 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.874449 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.874729 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.875650 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.876761 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.877950 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.878452 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.878578 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.881515 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.881524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.881538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.881574 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.883442 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.883911 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.883916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.883922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.884042 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.884343 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.884345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.884346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.884802 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.885002 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.885266 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.885600 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":133,"threads_started":1}
I20260925 22:54:03.888707 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.879428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:33077 (local address 127.0.0.1:40450)
0925 22:54:03.879804 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.879851 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.879965 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.880262 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.880267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.880653 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.880779 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.880786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.881611 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.881614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.882452 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.882466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.883105 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.883108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.883192 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.883768 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.883780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.885221 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.887022 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.887032 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.887038 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.887264 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.887556 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.887560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.887562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.887646 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.888031 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.888038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.888154 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.888321 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.888493 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
I20260925 22:54:03.888700 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.879521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.4.219.190:33077)
0925 22:54:03.879682 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.879688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.879936 (+ 248us) server_negotiation.cc:407] Connection header received
0925 22:54:03.880065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.880070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.880121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.880217 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.880907 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.881489 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.882611 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.882970 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.883088 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.885355 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.885358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.885361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.885384 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.886897 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.887371 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.887375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.887377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.887460 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.887734 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.887740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.887741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.887917 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.888031 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.888394 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.888501 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:54:03.890033 27456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:03.896982 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.888729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.4.219.190:33077)
0925 22:54:03.888930 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.888934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.888945 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:03.888978 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.888981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.889017 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.889093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.889816 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.890487 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.891283 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.891504 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.891628 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.893943 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.893949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.893955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.893984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.895359 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.895759 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.895763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.895765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.895848 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.896127 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.896130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.896132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.896338 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.896583 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.896691 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.896796 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260925 22:54:03.897836 27306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 3
I20260925 22:54:03.898111 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.890563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.4.219.189:46615)
0925 22:54:03.890697 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.890701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.890714 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:03.890759 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.890761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.890799 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.890902 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.891279 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.891836 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.892601 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.892913 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.893025 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.894774 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.894781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.894786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.894816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.896206 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.896687 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.896691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.896692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.896778 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.897184 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.897188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.897190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.897441 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.897545 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.897838 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.897967 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:54:03.898124 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.890430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:46615 (local address 127.0.0.1:47776)
0925 22:54:03.890541 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.890556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.890623 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.890911 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.890914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.891069 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.891154 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.891159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.891967 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.891972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.892469 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.892476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.893024 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.893028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.893116 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.893591 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.893602 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.894661 (+ 1059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.896312 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.896315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.896317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.896587 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.896881 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.896884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.896886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.896968 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.897570 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.897573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.897673 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.897765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.897951 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:54:03.898700 27456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:03.907878 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.898540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.4.219.188:45571)
0925 22:54:03.898821 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.898825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.898838 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:03.898869 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.898872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.898917 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.898994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.899403 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.900729 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.901656 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.901957 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902101 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.904420 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.904424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.904427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.904454 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.906098 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.906560 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.906563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.906565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.906645 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.906942 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.906949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.906954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.907145 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.907229 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.907555 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.907666 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":205}
I20260925 22:54:03.908025 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.898165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60846)
0925 22:54:03.898418 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.898431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.898498 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.899031 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.899037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.899203 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.899296 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.899303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.900837 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.900841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.901536 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.901546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.902109 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.902207 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.902662 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.902672 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.904229 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.906191 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.906194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.906196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.906451 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.906740 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.906744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.906746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.906838 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.907250 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.907253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.907371 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.907493 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.907866 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260925 22:54:03.909384 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.899292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.4.219.188:45571)
0925 22:54:03.899541 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.899550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.899587 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:54:03.899692 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.899695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.899725 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.899820 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.900143 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.901033 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.901804 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.902145 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902242 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.904912 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.904917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.904921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.904953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.906915 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.907388 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.907392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.907395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.907495 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.907813 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.907818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.907820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.908679 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.908774 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.909043 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.909187 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":205,"thread_start_us":127,"threads_started":1}
I20260925 22:54:03.909539 27460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.899127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60848)
0925 22:54:03.899533 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.899545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.899617 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.899839 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.899842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.899991 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.900043 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.900048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.901149 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.901152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.901691 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.901699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.902241 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.902313 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.902844 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.902859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.904748 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.907006 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.907012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.907016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.907283 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.907612 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.907615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.907617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.907724 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.908764 (+ 1040us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.908767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.908860 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.908982 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.909118 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3}
I20260925 22:54:03.910529 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.898264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.4.219.189:46615)
0925 22:54:03.898514 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.898519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.899194 (+ 675us) server_negotiation.cc:407] Connection header received
0925 22:54:03.899322 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.899325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.899363 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.899429 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.900273 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.901032 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902034 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.902363 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.902471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.906572 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.906578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.906585 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.906618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.908823 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.909264 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.909269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.909271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.909353 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.909654 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.909659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.909661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.909837 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.910063 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.910176 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.910295 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":176}
I20260925 22:54:03.910954 27381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } attempt: 3
I20260925 22:54:03.911547 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: Bootstrap starting.
I20260925 22:54:03.913801 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:03.914798 27306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:03.915727 27306 log.cc:824] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:03.914913 27280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1af20b4b3a924243828e20a910ce904a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40456
Sep 25 22:54:03 dist-test-slave-kb1g krb5kdc[27225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
I20260925 22:54:03.918823 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: No bootstrap required, opened a new log
I20260925 22:54:03.919672 27456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: Bootstrap starting.
I20260925 22:54:03.921228 27384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.911293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:45571 (local address 127.0.0.1:60854)
0925 22:54:03.911423 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:03.911439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:03.911512 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:03.912320 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:03.912323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:03.912497 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:03.912636 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.912643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.913827 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.913831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.914672 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:03.914687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.915351 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.915355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:03.915451 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.915941 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:03.915957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:03.917394 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:03.919529 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.919532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.919534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.919796 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.920090 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:03.920093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:03.920095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:03.920185 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:03.920668 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:03.920671 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:03.920768 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:54:03.920885 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:54:03.921062 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 22:54:03.921255 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.911655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.4.219.188:45571)
0925 22:54:03.912093 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.912097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.912111 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:03.912145 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.912147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.912186 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.912752 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.912854 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.913676 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.914857 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.915221 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.915339 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.917539 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.917542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.917545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.917571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.919399 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.919886 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.919889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.919891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.919982 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.920289 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.920292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.920295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.920512 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.920661 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.920959 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.921098 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260925 22:54:03.922359 27306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.922581 27306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af20b4b3a924243828e20a910ce904a, State: Initialized, Role: FOLLOWER
I20260925 22:54:03.922899 27306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.923409 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.923617 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.924227 27306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:03.924335 27280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:54:03.924427 27381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: Bootstrap starting.
I20260925 22:54:03.924449 27280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:03.925683 27280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.927045 27381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:03.926834 27280 leader_election.cc:288] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [CANDIDATE]: Term 1 election: Requested vote from peers e33ee51f5cc647f5b7eed3ff7a465219 (127.4.219.189:46615), 2dc4d452d0384346bdbabaff61316dbd (127.4.219.188:45571)
I20260925 22:54:03.927903 27381 log.cc:824] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:03.929142 27456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: Neither blocks nor log segments found. Creating new log.
W20260925 22:54:03.929898 27475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:03.930020 27475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:03.930358 27456 log.cc:824] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:03.932493 27456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: No bootstrap required, opened a new log
I20260925 22:54:03.931401 27238 master_runner.cc:186] Error getting master registration for 127.4.219.189:46615: OK, instance_id { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" instance_seqno: 1790376843657658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:54:03.934648 27238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:54:03.936596 27381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: No bootstrap required, opened a new log
I20260925 22:54:03.936626 27456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.936865 27456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc4d452d0384346bdbabaff61316dbd, State: Initialized, Role: FOLLOWER
I20260925 22:54:03.937198 27456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.938464 27456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:03.938542 27430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1af20b4b3a924243828e20a910ce904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2dc4d452d0384346bdbabaff61316dbd"
I20260925 22:54:03.938745 27430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:03.940689 27430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af20b4b3a924243828e20a910ce904a in term 1.
I20260925 22:54:03.941550 27254 leader_election.cc:302] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1af20b4b3a924243828e20a910ce904a, 2dc4d452d0384346bdbabaff61316dbd; no voters:
I20260925 22:54:03.941886 27464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:03.942106 27464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [term 1 LEADER]: Becoming Leader. State: Replica: 1af20b4b3a924243828e20a910ce904a, State: Running, Role: LEADER
I20260925 22:54:03.942416 27464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.944155 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af20b4b3a924243828e20a910ce904a. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.944284 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:03.944463 27381 raft_consensus.cc:348] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.944972 27381 raft_consensus.cc:729] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33ee51f5cc647f5b7eed3ff7a465219, State: Initialized, Role: FOLLOWER
I20260925 22:54:03.945716 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.945976 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.946728 27381 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } }
I20260925 22:54:03.948982 27355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1af20b4b3a924243828e20a910ce904a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e33ee51f5cc647f5b7eed3ff7a465219"
I20260925 22:54:03.949198 27355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:03.949992 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.931861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.4.219.188:45571)
0925 22:54:03.931977 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.931983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.940686 (+ 8703us) server_negotiation.cc:407] Connection header received
0925 22:54:03.940884 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.940889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.940940 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.940991 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.942115 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.942826 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.943935 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.944251 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.944334 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.946370 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.946374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.946377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.946399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.948138 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.948628 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.948632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.948634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.948714 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.949030 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.949033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.949034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.949232 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.949379 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.949721 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.949830 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:54:03.950712 27355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1af20b4b3a924243828e20a910ce904a in term 1.
I20260925 22:54:03.952610 27381 sys_catalog.cc:564] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:03.950510 27387 master_runner.cc:186] Error getting master registration for 127.4.219.189:46615: OK, instance_id { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" instance_seqno: 1790376843657658 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:54:03.954447 27387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:54:03.956745 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.937366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.4.219.189:46615)
0925 22:54:03.937514 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.937519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.945685 (+ 8166us) server_negotiation.cc:407] Connection header received
0925 22:54:03.945887 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.945893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.945946 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.946002 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.946910 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.947759 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.949031 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.949403 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.949496 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.951571 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.951575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.951578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.951602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.953246 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.954962 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.954965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.954967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.955056 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.955687 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.955690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.955694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.955933 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.956041 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.956420 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.956569 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:54:03.958600 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:03.936938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.4.219.190:33077)
0925 22:54:03.937046 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:03.937051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:03.944803 (+ 7752us) server_negotiation.cc:407] Connection header received
0925 22:54:03.945013 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:03.945018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:03.945074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:03.945137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:03.950703 (+ 5566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.951501 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.952654 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:03.953001 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:03.953089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:03.954957 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:03.954960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:03.954963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:03.954987 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:03.956727 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.957257 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.957262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.957266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.957353 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:03.957659 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:03.957662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:03.957664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:03.957874 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:03.958009 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:03.958331 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 22:54:03.958449 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:54:03.962314 27499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 22:54:03.963196 27512 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:03.963281 27512 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:03.963356 27499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:54:03.965674 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.965804 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 22:54:03.967227 27500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:03.967340 27500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:03.968750 27355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [term 1 FOLLOWER]: Refusing update from remote peer 1af20b4b3a924243828e20a910ce904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:03.969332 27430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [term 1 FOLLOWER]: Refusing update from remote peer 1af20b4b3a924243828e20a910ce904a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:03.969475 27497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:03.969929 27464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:03.973287 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af20b4b3a924243828e20a910ce904a. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.973862 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.973878 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af20b4b3a924243828e20a910ce904a. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.973995 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.979732 27514 mvcc.cc:205] Tried to move back new op lower bound from 7333383552889012224 to 7333383552791633920. Current Snapshot: MvccSnapshot[applied={T|T < 7333383552889012224}]
I20260925 22:54:03.980221 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.980366 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:03.981483 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.981617 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.985643 27464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.985746 27464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:03.986847 27485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1af20b4b3a924243828e20a910ce904a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1af20b4b3a924243828e20a910ce904a" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 33077 } } peers { permanent_uuid: "e33ee51f5cc647f5b7eed3ff7a465219" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 46615 } } peers { permanent_uuid: "2dc4d452d0384346bdbabaff61316dbd" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 45571 } } }
I20260925 22:54:03.986948 27485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:03.987637 27517 mvcc.cc:205] Tried to move back new op lower bound from 7333383552889012224 to 7333383552791633920. Current Snapshot: MvccSnapshot[applied={T|T < 7333383552889012224}]
I20260925 22:54:03.987860 27516 mvcc.cc:205] Tried to move back new op lower bound from 7333383552889012224 to 7333383552791633920. Current Snapshot: MvccSnapshot[applied={T|T < 7333383552889012224}]
I20260925 22:54:03.989532 27499 catalog_manager.cc:1347] Generated new cluster ID: 6d9d1c6940b54ae9ad181a8a65f5cdc7
I20260925 22:54:03.989601 27499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:04.013446 27499 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:04.014853 27499 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:04.041596 27499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af20b4b3a924243828e20a910ce904a: Generated new TSK 0
I20260925 22:54:04.042589 27499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:54:04.078868 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.075685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.4.219.190:33077)
0925 22:54:04.075825 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.075831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.077184 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 22:54:04.077890 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.077894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.077952 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.078010 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.078713 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40482: BlockingRecv error: recv got EOF from 127.0.0.1:40482 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:54:04.080523 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.076476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.4.219.189:46615)
0925 22:54:04.076638 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.076643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.079321 (+ 2678us) server_negotiation.cc:407] Connection header received
0925 22:54:04.079691 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.079695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.079752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.079810 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.080395 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47814: BlockingRecv error: recv got EOF from 127.0.0.1:47814 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 22:54:04.081871 27462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.077554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.4.219.188:45571)
0925 22:54:04.077687 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.077694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.080917 (+ 3223us) server_negotiation.cc:407] Connection header received
0925 22:54:04.081099 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.081102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.081154 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.081207 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.081744 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60870: BlockingRecv error: recv got EOF from 127.0.0.1:60870 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:54:04.086858 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.084582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.4.219.189:46615)
0925 22:54:04.084697 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.084702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.085872 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 22:54:04.086102 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.086105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.086181 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.086237 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.086747 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47820: BlockingRecv error: recv got EOF from 127.0.0.1:47820 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:54:04.092311 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.090437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.4.219.189:46615)
0925 22:54:04.090538 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.090543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.091263 (+ 720us) server_negotiation.cc:407] Connection header received
0925 22:54:04.091467 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.091471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.091544 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.091597 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.092183 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47824: BlockingRecv error: recv got EOF from 127.0.0.1:47824 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 22:54:04.097913 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.096336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.4.219.189:46615)
0925 22:54:04.096435 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.096439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.097024 (+ 585us) server_negotiation.cc:407] Connection header received
0925 22:54:04.097209 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.097212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.097260 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.097330 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.097819 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47836: BlockingRecv error: recv got EOF from 127.0.0.1:47836 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 22:54:04.105641 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.103956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.4.219.189:46615)
0925 22:54:04.104095 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.104100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.104691 (+ 591us) server_negotiation.cc:407] Connection header received
0925 22:54:04.104893 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.104896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.104948 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.105001 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.105538 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47838: BlockingRecv error: recv got EOF from 127.0.0.1:47838 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 22:54:04.118100 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.116174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.4.219.189:46615)
0925 22:54:04.116311 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.116315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.117006 (+ 691us) server_negotiation.cc:407] Connection header received
0925 22:54:04.117240 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.117244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.117315 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.117375 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.117988 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47844: BlockingRecv error: recv got EOF from 127.0.0.1:47844 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:54:04.138963 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.136895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.4.219.189:46615)
0925 22:54:04.137054 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.137061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.137776 (+ 715us) server_negotiation.cc:407] Connection header received
0925 22:54:04.138018 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.138022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.138083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.138144 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.138811 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47850: BlockingRecv error: recv got EOF from 127.0.0.1:47850 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:54:04.175360 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.173784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.4.219.189:46615)
0925 22:54:04.173922 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.173927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.174198 (+ 271us) server_negotiation.cc:407] Connection header received
0925 22:54:04.174438 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.174442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.174500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.174556 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.175240 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47856: BlockingRecv error: recv got EOF from 127.0.0.1:47856 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 22:54:04.242740 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.241533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.4.219.189:46615)
0925 22:54:04.241712 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.241716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.241737 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:54:04.241906 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.241909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.241962 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.242037 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.242600 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47870: BlockingRecv error: recv got EOF from 127.0.0.1:47870 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260925 22:54:04.374253 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.372982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.4.219.189:46615)
0925 22:54:04.373172 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.373177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.373368 (+ 191us) server_negotiation.cc:407] Connection header received
0925 22:54:04.373521 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.373524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.373581 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.373674 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.374137 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47876: BlockingRecv error: recv got EOF from 127.0.0.1:47876 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 22:54:04.633546 27385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:04.632463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47884 (local address 127.4.219.189:46615)
0925 22:54:04.632650 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:04.632654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:04.632671 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:54:04.632813 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:04.632816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:04.632870 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:04.632955 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:04.633416 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47884: BlockingRecv error: recv got EOF from 127.0.0.1:47884 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:54:04.964814 27512 catalog_manager.cc:1259] Loaded cluster ID: 6d9d1c6940b54ae9ad181a8a65f5cdc7
I20260925 22:54:04.964900 27512 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: loading cluster ID for follower catalog manager: success
I20260925 22:54:04.966614 27512 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: acquiring CA information for follower catalog manager: success
I20260925 22:54:04.967780 27512 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e33ee51f5cc647f5b7eed3ff7a465219: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:54:04.969595 27500 catalog_manager.cc:1259] Loaded cluster ID: 6d9d1c6940b54ae9ad181a8a65f5cdc7
I20260925 22:54:04.969777 27500 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: loading cluster ID for follower catalog manager: success
I20260925 22:54:04.971293 27500 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: acquiring CA information for follower catalog manager: success
I20260925 22:54:04.972348 27500 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2dc4d452d0384346bdbabaff61316dbd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:54:05.151710 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:05.147696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.4.219.189:46615)
0925 22:54:05.148009 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:05.148014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:05.148030 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:54:05.148188 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:05.148191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:05.148318 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:05.148442 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:05.148736 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.149274 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.150159 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.150394 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.150502 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:05.150824 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:05.151287 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:05.151439 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:05.151487 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 22:54:05.151543 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260925 22:54:05.159253 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:05.154977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.4.219.188:45571)
0925 22:54:05.155202 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:05.155206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:05.155219 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:54:05.155424 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:05.155426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:05.155547 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:05.155646 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:05.155858 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.156385 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.157258 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.157594 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.157715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:05.158156 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:05.158841 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:05.158980 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:05.159016 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 22:54:05.159070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260925 22:54:05.167827 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:05.163044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.4.219.188:45571)
0925 22:54:05.163160 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:05.163165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:05.163443 (+ 278us) server_negotiation.cc:407] Connection header received
0925 22:54:05.163615 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:05.163618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:05.163668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:05.163712 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:05.164315 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.165156 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.166607 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.166927 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.167015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:05.167352 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:05.167430 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:05.167487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:05.167626 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 22:54:05.167679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:54:05.168608 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:05.162952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.4.219.190:33077)
0925 22:54:05.163246 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:05.163258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:05.163576 (+ 318us) server_negotiation.cc:407] Connection header received
0925 22:54:05.163675 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:05.163678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:05.163845 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:05.163971 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:05.164353 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.165296 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.166547 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.166962 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.167073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:05.167556 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:05.168117 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:05.168199 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:05.168350 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:54:05.168409 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":56,"threads_started":1}
I20260925 22:54:05.169938 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:05.163559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.4.219.189:46615)
0925 22:54:05.163880 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:05.163885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:05.164251 (+ 366us) server_negotiation.cc:407] Connection header received
0925 22:54:05.165462 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:05.165466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:05.165523 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:05.165569 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:05.166004 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.166725 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.167825 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:05.168208 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:05.168337 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:05.168914 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:05.168993 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:05.169080 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:05.169728 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 22:54:05.169804 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260925 22:54:05.172253 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27238
I20260925 22:54:05.178851 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27313
I20260925 22:54:05.184937 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27387
2026-09-25T22:54:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3105 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:54:05 dist-test-slave-kb1g krb5kdc[27605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:54:05 dist-test-slave-kb1g krb5kdc[27605](info): set up 2 sockets
Sep 25 22:54:05 dist-test-slave-kb1g krb5kdc[27605](info): commencing operation
krb5kdc: starting...
W20260925 22:54:06.216959 4974 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/test-user.keytab.
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.219.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
Entry for principal kudu/127.4.219.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab.
I20260925 22:54:06.343521 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44781
--webserver_interface=127.4.219.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:54:06.454571 27618 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:06.454896 27618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:06.454945 27618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:06.455014 27618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:54:06.459041 27618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:06.459118 27618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:06.459156 27618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:06.459192 27618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:06.459223 27618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:06.464368 27618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.190.keytab
--principal=kudu/127.4.219.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.190:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.219.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:06.465430 27618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:06.466459 27618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:06.475134 27623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.475266 27626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.475170 27624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:06.475159 27618 server_base.cc:1034] running on GCE node
I20260925 22:54:06.476001 27618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:06.477128 27618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:06.478327 27618 hybrid_clock.cc:648] HybridClock initialized: now 1790376846478310 us; error 25 us; skew 500 ppm
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:06.485917 27618 init.cc:377] Logged in from keytab as kudu/127.4.219.190@KRBTEST.COM (short username kudu)
I20260925 22:54:06.486976 27618 webserver.cc:460] Webserver started at http://127.4.219.190:36569/ using document root <none> and password file <none>
I20260925 22:54:06.487591 27618 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:06.487690 27618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:06.487951 27618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:06.490200 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data/instance:
uuid: "68f05abc8ef3459e88b466ea711ed536"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "2552f9a7b3fb566ad7cd37523df6dcd1"
server_key_iv: "6ddc6bbf84e39d72219f1bc500a01878"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.490797 27618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance:
uuid: "68f05abc8ef3459e88b466ea711ed536"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "2552f9a7b3fb566ad7cd37523df6dcd1"
server_key_iv: "6ddc6bbf84e39d72219f1bc500a01878"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.494616 27618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:54:06.497113 27633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:06.498370 27618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:06.498521 27618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal
uuid: "68f05abc8ef3459e88b466ea711ed536"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "2552f9a7b3fb566ad7cd37523df6dcd1"
server_key_iv: "6ddc6bbf84e39d72219f1bc500a01878"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.498637 27618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:06.513643 27618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:06.516573 27618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:06.516778 27618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:06.523777 27618 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.190:44781
I20260925 22:54:06.524071 27685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.190:44781 every 8 connection(s)
I20260925 22:54:06.524772 27618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/data/info.pb
I20260925 22:54:06.527558 27686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:06.529367 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27618
I20260925 22:54:06.529480 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-0/wal/instance
I20260925 22:54:06.529767 4974 external_mini_cluster.cc:1428] Setting key 0f78d38d99d17c40fde71d7817dcf6fb
I20260925 22:54:06.532282 27686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.219.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.189@KRBTEST.COM" created.
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:54:06.548112 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.534643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44781 (local address 127.0.0.1:55826)
0925 22:54:06.535249 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.535307 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.536134 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.536985 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.536989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.537357 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.537613 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.537625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.538905 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.538909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.539623 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.539631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.540293 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.540297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.540451 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.541034 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.541058 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.543007 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.545298 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.545306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.545313 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.545525 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.545911 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.545913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.545914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.546014 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.546779 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.546783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.546900 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.547038 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.547273 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":3,"spinlock_wait_cycles":54528,"thread_start_us":147,"threads_started":1}
I20260925 22:54:06.548117 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.534643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.4.219.190:44781)
0925 22:54:06.535161 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.535182 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.535434 (+ 252us) server_negotiation.cc:407] Connection header received
0925 22:54:06.536454 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.536461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.536747 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.536970 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.537773 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.538769 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.539768 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.540141 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.540346 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.543152 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.543157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.543167 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.543188 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.545192 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.545641 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.545650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.545653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.545799 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.546109 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.546115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.546118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.546647 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.546784 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.547146 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.547350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":335,"spinlock_wait_cycles":3328,"thread_start_us":144,"threads_started":1}
I20260925 22:54:06.550413 27686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:06.551134 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.550875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34684)
0925 22:54:06.550988 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.551055 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:54:06.551359 27637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
I20260925 22:54:06.552137 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.551927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34690)
0925 22:54:06.552037 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.552090 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:54:06.552269 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:41973: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
Entry for principal kudu/127.4.219.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab.
I20260925 22:54:06.557281 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:41973
--webserver_interface=127.4.219.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:06.601513 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } attempt: 1
I20260925 22:54:06.602432 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.602095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34702)
0925 22:54:06.602264 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.602340 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:54:06.603145 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.602930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34704)
0925 22:54:06.603033 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.603084 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:54:06.603295 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:41973: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
W20260925 22:54:06.667414 27693 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:06.667735 27693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:06.667817 27693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:06.667899 27693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:54:06.671485 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } attempt: 2
W20260925 22:54:06.672116 27693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:06.672174 27693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:06.672200 27693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:06.672231 27693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:06.672263 27693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:06.672415 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.672010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34720)
0925 22:54:06.672239 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.672327 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 22:54:06.673168 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.672948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34732)
0925 22:54:06.673052 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.673106 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 22:54:06.673329 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.189:41973: Network error: Client connection negotiation failed: client connection to 127.4.219.189:41973: connect: Connection refused (error 111)
I20260925 22:54:06.677819 27693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.189.keytab
--principal=kudu/127.4.219.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.189:41973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.219.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:06.678851 27693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:06.679853 27693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:54:06.688385 27698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.688380 27699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.688385 27701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:06.688651 27693 server_base.cc:1034] running on GCE node
I20260925 22:54:06.689208 27693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:06.690316 27693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:06.691519 27693 hybrid_clock.cc:648] HybridClock initialized: now 1790376846691502 us; error 29 us; skew 500 ppm
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:06.694358 27693 init.cc:377] Logged in from keytab as kudu/127.4.219.189@KRBTEST.COM (short username kudu)
I20260925 22:54:06.695230 27693 webserver.cc:460] Webserver started at http://127.4.219.189:34013/ using document root <none> and password file <none>
I20260925 22:54:06.695784 27693 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:06.695873 27693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:06.696100 27693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:06.698150 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data/instance:
uuid: "6336d051c0654888b4c5918b03a58ded"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "139549d9ba875b5a48233f0a956ea34d"
server_key_iv: "afc75dc8fe7ef288234ad11ef49edb4c"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.698724 27693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal/instance:
uuid: "6336d051c0654888b4c5918b03a58ded"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "139549d9ba875b5a48233f0a956ea34d"
server_key_iv: "afc75dc8fe7ef288234ad11ef49edb4c"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.702414 27693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:54:06.705148 27708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:06.706434 27693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 22:54:06.706573 27693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal
uuid: "6336d051c0654888b4c5918b03a58ded"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "139549d9ba875b5a48233f0a956ea34d"
server_key_iv: "afc75dc8fe7ef288234ad11ef49edb4c"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.706688 27693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:06.731422 27693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:06.734750 27693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:06.734959 27693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:06.742113 27693 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.189:41973
I20260925 22:54:06.742511 27760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.189:41973 every 8 connection(s)
I20260925 22:54:06.743098 27693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/data/info.pb
I20260925 22:54:06.743325 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27693
I20260925 22:54:06.743463 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-1/wal/instance
I20260925 22:54:06.743809 4974 external_mini_cluster.cc:1428] Setting key 39bf63f390ad717062091520bf448967
I20260925 22:54:06.745995 27761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:06.749637 27761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.219.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.219.188@KRBTEST.COM" created.
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:54:06.764148 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.751470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.4.219.190:44781)
0925 22:54:06.751604 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.751609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.752592 (+ 983us) server_negotiation.cc:407] Connection header received
0925 22:54:06.753628 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.753631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.753680 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.753770 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.755065 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.755974 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.757172 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.757439 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.757587 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.760458 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.760461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.760463 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.760488 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.761997 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.762600 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.762603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.762604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.762672 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.763060 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.763062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.763063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.763229 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.763304 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.763911 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.764019 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:54:06.764513 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } attempt: 3
I20260925 22:54:06.765125 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.751872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44781 (local address 127.0.0.1:55842)
0925 22:54:06.752362 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.752421 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.753417 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.753921 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.753928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.754347 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.754906 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.754920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.756108 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.756110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.757044 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.757053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.757623 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.757627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.757802 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.758431 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.758448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.760272 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.762123 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.762132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.762151 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.762498 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.762776 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.762779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.762783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.762965 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.763349 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.763354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.763600 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.763835 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.764098 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260925 22:54:06.766639 27761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
Entry for principal kudu/127.4.219.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab.
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:06.772145 4974 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskre8WLI/build/debug/bin/kudu
/tmp/dist-test-taskre8WLI/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:37879
--webserver_interface=127.4.219.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.219.148:36789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:54:06.775096 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.764963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34746)
0925 22:54:06.765102 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.765116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.765214 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.765989 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.765993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.766163 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.766315 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.766321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.767605 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.767608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.768379 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.768395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.769152 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.769156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.769277 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.769825 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.769837 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.771219 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.773393 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.773397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.773399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.773624 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.773972 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.773975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.773978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.774067 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.774522 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.774526 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.774625 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.774753 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.774899 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:54:06.775810 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.765063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.4.219.189:41973)
0925 22:54:06.765586 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.765601 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.765627 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:54:06.765673 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.765679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.765854 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.766002 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.766427 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.767491 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.768493 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.769041 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.769152 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.771369 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.771382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.771399 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.771433 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.773265 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.773755 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.773764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.773768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.773872 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.774166 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.774169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.774171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.774414 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.774558 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.775371 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.775542 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":415,"thread_start_us":149,"threads_started":1}
I20260925 22:54:06.776856 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.767120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34760)
0925 22:54:06.767210 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.767222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.767289 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.767751 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.767753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.767884 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.767998 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.768006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.768901 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.768903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.769535 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.769545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.770038 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.770041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.770128 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.770564 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.770575 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.771857 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.774714 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.774720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.774725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.774964 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.775506 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.775509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.775510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.775589 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.776227 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.776232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.776337 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.776460 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.776649 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 22:54:06.776969 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.767215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.4.219.189:41973)
0925 22:54:06.767528 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.767535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.767564 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:54:06.767601 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.767603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.767643 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.767729 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.768117 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.768790 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.769667 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.769911 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.770015 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.772024 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.772027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.772029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.772051 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.774609 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.775064 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.775070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.775073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.775396 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.775682 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.775685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.775686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.775870 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.776210 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.776523 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.776733 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260925 22:54:06.778038 27686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:06.778038 27761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:06.778793 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.778531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36082)
0925 22:54:06.778652 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.778729 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:54:06.778843 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.778554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36084)
0925 22:54:06.778723 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.778765 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260925 22:54:06.779049 27711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
I20260925 22:54:06.779516 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.779276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36090)
0925 22:54:06.779409 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.779458 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 22:54:06.779641 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
I20260925 22:54:06.780099 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.779936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36104)
0925 22:54:06.780017 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.780056 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 22:54:06.780252 27761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
I20260925 22:54:06.810801 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 1
I20260925 22:54:06.811645 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.811353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36108)
0925 22:54:06.811505 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.811567 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:54:06.812732 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.812492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36124)
0925 22:54:06.812630 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.812678 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 22:54:06.812870 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
I20260925 22:54:06.829488 27761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 1
I20260925 22:54:06.830397 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.830044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36138)
0925 22:54:06.830213 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.830318 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 22:54:06.831259 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.830984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36154)
0925 22:54:06.831108 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.831167 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:54:06.831409 27761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
W20260925 22:54:06.885200 27768 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:54:06.885519 27768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:54:06.885581 27768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:54:06.885658 27768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:54:06.888027 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 2
I20260925 22:54:06.888942 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.888630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36158)
0925 22:54:06.888784 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.888861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 22:54:06.889744 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.889524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36170)
0925 22:54:06.889637 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.889691 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:54:06.889890 27686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
W20260925 22:54:06.889931 27768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:54:06.889977 27768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:54:06.890036 27768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:54:06.890091 27768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:54:06.890125 27768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:54:06.895377 27768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.219.148:36789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.219.190:44781,127.4.219.189:41973,127.4.219.188:37879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/krb5kdc/kudu_127.4.219.188.keytab
--principal=kudu/127.4.219.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.219.188:37879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.219.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:54:06.896407 27768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:54:06.897432 27768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:54:06.899600 27761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 2
I20260925 22:54:06.900394 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.900099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36184)
0925 22:54:06.900259 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.900324 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:54:06.901204 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.900972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36192)
0925 22:54:06.901103 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.901153 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 22:54:06.901366 27761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.219.188:37879: Network error: Client connection negotiation failed: client connection to 127.4.219.188:37879: connect: Connection refused (error 111)
W20260925 22:54:06.906045 27775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.906047 27777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:54:06.906045 27774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:54:06.906373 27768 server_base.cc:1034] running on GCE node
I20260925 22:54:06.906867 27768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:54:06.907949 27768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:54:06.909097 27768 hybrid_clock.cc:648] HybridClock initialized: now 1790376846909083 us; error 25 us; skew 500 ppm
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:54:06.911810 27768 init.cc:377] Logged in from keytab as kudu/127.4.219.188@KRBTEST.COM (short username kudu)
I20260925 22:54:06.912732 27768 webserver.cc:460] Webserver started at http://127.4.219.188:42605/ using document root <none> and password file <none>
I20260925 22:54:06.913267 27768 fs_manager.cc:359] Metadata directory not provided
I20260925 22:54:06.913324 27768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:54:06.913528 27768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:54:06.915469 27768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data/instance:
uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "a0004d0336588ea7c03b4f2e8a787bfe"
server_key_iv: "52ddb5e887f3597f6bf14848c5ee25be"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.916052 27768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal/instance:
uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "a0004d0336588ea7c03b4f2e8a787bfe"
server_key_iv: "52ddb5e887f3597f6bf14848c5ee25be"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.919668 27768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260925 22:54:06.922071 27784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:54:06.923175 27768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 22:54:06.923303 27768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data,/tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal
uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d"
format_stamp: "Formatted at 2026-09-25 22:54:06 on dist-test-slave-kb1g"
server_key: "a0004d0336588ea7c03b4f2e8a787bfe"
server_key_iv: "52ddb5e887f3597f6bf14848c5ee25be"
server_key_version: "encryptionkey@0"
I20260925 22:54:06.923405 27768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:54:06.948665 27768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:54:06.951754 27768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:54:06.951972 27768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:54:06.959018 27768 rpc_server.cc:307] RPC server started. Bound to: 127.4.219.188:37879
I20260925 22:54:06.959295 27836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.219.188:37879 every 8 connection(s)
I20260925 22:54:06.959967 27768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/data/info.pb
I20260925 22:54:06.960728 4974 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskre8WLI/build/debug/bin/kudu as pid 27768
I20260925 22:54:06.960867 4974 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskre8WLI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376566921023-4974-0/minicluster-data/master-2/wal/instance
I20260925 22:54:06.961162 4974 external_mini_cluster.cc:1428] Setting key 8a2a67291c72a48dea116504a05251d4
I20260925 22:54:06.963315 27837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:54:06.966890 27837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
I20260925 22:54:06.975625 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.962460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.4.219.188:37879)
0925 22:54:06.962869 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.962884 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.962921 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:54:06.963612 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.963629 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.963959 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.964288 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.965149 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.966153 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.967020 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.967538 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.967710 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.970579 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.970585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.970598 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.970635 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.972436 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.972962 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.972970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.972976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.973092 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.973355 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.973358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.973360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.973709 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.973893 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.974166 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.974577 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260925 22:54:06.977617 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.968523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.4.219.190:44781)
0925 22:54:06.968669 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.968674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.969002 (+ 328us) server_negotiation.cc:407] Connection header received
0925 22:54:06.969169 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.969171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.969215 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.969283 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.970000 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.970567 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.971707 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.972049 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.972153 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.974359 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.974363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.974366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.974397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.975868 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.976304 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.976308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.976309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.976403 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.976722 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.976726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.976728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.976905 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.977027 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.977358 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.977459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:54:06.977625 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.968452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.190:44781 (local address 127.0.0.1:55854)
0925 22:54:06.968874 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.968932 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.969063 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.969362 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.969371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.969745 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.969874 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.969880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.970691 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.970694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.971576 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.971589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.972166 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.972169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.972247 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.972717 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.972733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.974217 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.975973 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.975982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.975988 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.976196 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.976499 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.976502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.976504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.976635 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.977027 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.977035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.977144 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.977271 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.977454 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
I20260925 22:54:06.978612 27837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.190@KRBTEST.COM
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:06.986604 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.977745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.4.219.190:44781)
0925 22:54:06.978014 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.978018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.978029 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:54:06.978160 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.978163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.978205 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.978290 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.978985 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.979541 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.980289 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.980504 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.980658 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.983138 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.983141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.983143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.983162 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.984848 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985273 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.985278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.985283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.985362 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985645 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.985650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.985653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.985845 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.986008 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.986330 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.986441 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":211}
I20260925 22:54:06.986624 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.979172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.4.219.189:41973)
0925 22:54:06.979311 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.979318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.979334 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:54:06.979377 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.979380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.979436 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.979519 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.979878 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.980378 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.981162 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.981424 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.981532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.983618 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.983622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.983624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.983650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.984931 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985313 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.985316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.985317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.985386 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985702 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.985704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.985706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.985868 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.986083 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.986378 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.986491 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:54:06.986966 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.979067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.189:41973 (local address 127.0.0.1:34768)
0925 22:54:06.979196 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.979227 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.979293 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.979530 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.979533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.979690 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.979767 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.979772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.980488 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.980495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.981047 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.981055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.981530 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.981533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.981623 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.982073 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.982093 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.983473 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.985026 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.985030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.985198 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.985503 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.985505 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.985507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.985582 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.986095 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.986097 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.986168 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.986745 (+ 577us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.986866 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 22:54:06.987937 27837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:54:06.989068 27686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 3
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.219.189@KRBTEST.COM
I20260925 22:54:06.992511 27761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } attempt: 3
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.188@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.190@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
I20260925 22:54:06.996762 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.987825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.4.219.189:41973)
0925 22:54:06.987928 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.987932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.987972 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:54:06.988098 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.988101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.988139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.988226 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.988949 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.989589 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.990352 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.990673 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.990776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.993103 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.993106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.993109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.993133 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.994900 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.995335 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.995338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.995340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.995421 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.995703 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.995707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.995708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.995902 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.996089 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.996417 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.996593 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 22:54:06 dist-test-slave-kb1g krb5kdc[27605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.219.189@KRBTEST.COM for kudu/127.4.219.188@KRBTEST.COM
I20260925 22:54:06.999266 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.988408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36204)
0925 22:54:06.988492 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.988506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.988624 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.988837 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.988839 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.988995 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.989078 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.989088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.990832 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.990835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.991606 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.991615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.992213 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.992216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.992284 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.992772 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.992785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.994147 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.997461 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.997466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.997470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.997719 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.997996 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.998000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.998005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.998089 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:06.998697 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:06.998699 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:06.998782 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 22:54:06.998877 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:54:06.999087 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 22:54:06.999620 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.988519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.4.219.188:37879)
0925 22:54:06.988651 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.988657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.988669 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:54:06.988699 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.988702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.988732 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.988831 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.989198 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.990724 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.991717 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.992029 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.992107 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.994269 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.994272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.994275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.994298 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.997354 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.997818 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.997821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.997822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.997901 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.998184 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.998188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.998193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.998575 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:06.998664 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:06.999008 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 22:54:06.999484 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":376,"server-negotiator.queue_time_us":73}
I20260925 22:54:07.001151 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.989421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36206)
0925 22:54:06.989514 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.989529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.989597 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.990656 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.990659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.990837 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.990930 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.990939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.992205 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.992209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.993314 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.993328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.994734 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.994741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.994858 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.995467 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.995482 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.997024 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:06.999356 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:06.999359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:06.999362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:06.999610 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:07.000035 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:07.000038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:07.000039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:07.000139 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:07.000637 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:07.000640 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:07.000750 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 22:54:07.000841 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:54:07.000975 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 22:54:07.001611 4974 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:54:07.001732 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.989483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.4.219.188:37879)
0925 22:54:06.990471 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.990477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.990486 (+ 9us) server_negotiation.cc:407] Connection header received
0925 22:54:06.990511 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.990513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.990543 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.990640 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.991328 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.992091 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.993745 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.994374 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.996023 (+ 1649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.997156 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.997159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.997161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.997180 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:06.999242 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:06.999852 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:06.999855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:06.999856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:06.999931 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.000245 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.000248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.000250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.000470 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:07.001320 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:07.001406 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 22:54:07.001537 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":137,"threads_started":1}
I20260925 22:54:07.003494 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.993329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.4.219.188:37879)
0925 22:54:06.993656 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:06.993667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:06.993682 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:54:06.993717 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:06.993721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:06.993767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:06.994319 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:06.994450 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.995221 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.996261 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.996632 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.997050 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.998842 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:06.998847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:06.998850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:06.998875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:07.001796 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.002315 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.002317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.002318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.002382 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.002658 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.002663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.002665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.002879 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:07.002989 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:07.003233 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:54:07.003335 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":133,"threads_started":1}
I20260925 22:54:07.003489 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:06.992902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.219.188:37879 (local address 127.0.0.1:36220)
0925 22:54:06.993122 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 22:54:06.993135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:54:06.993196 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:54:06.993912 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:54:06.993917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:54:06.994085 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:54:06.994200 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.994210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.995340 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.995344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.996133 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:54:06.996141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:54:06.996763 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:54:06.996766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:54:06.996859 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:06.997286 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:54:06.997301 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:54:06.998697 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:54:07.001928 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:07.001935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:07.001938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:07.002192 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:07.002479 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:54:07.002484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:54:07.002486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:54:07.002570 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:54:07.002981 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:54:07.002984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:54:07.003080 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 22:54:07.003183 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 22:54:07.003335 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260925 22:54:07.003515 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: Bootstrap starting.
I20260925 22:54:07.004863 27686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: Bootstrap starting.
I20260925 22:54:07.006032 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: Bootstrap starting.
I20260925 22:54:07.005223 27660 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "68f05abc8ef3459e88b466ea711ed536"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864
I20260925 22:54:07.007611 27686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:07.008102 27761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:07.008517 27686 log.cc:824] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:07.008659 27837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: Neither blocks nor log segments found. Creating new log.
I20260925 22:54:07.008898 27761 log.cc:824] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:07.009677 27837 log.cc:824] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:54:07.010890 27686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: No bootstrap required, opened a new log
I20260925 22:54:07.011626 27837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: No bootstrap required, opened a new log
I20260925 22:54:07.013964 27686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.014189 27686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f05abc8ef3459e88b466ea711ed536, State: Initialized, Role: FOLLOWER
I20260925 22:54:07.014520 27686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.015021 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.015161 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.015944 27686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:07.015978 27660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:54:07.016112 27660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:07.016027 27837 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.016247 27837 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9f3acdfe004a0ebdfd071ff7e3d77d, State: Initialized, Role: FOLLOWER
I20260925 22:54:07.016629 27837 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.017450 27660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.017973 27837 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:54:07.018250 27639 tablet.cc:2369] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:54:07.018546 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: No bootstrap required, opened a new log
I20260925 22:54:07.019500 27811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68f05abc8ef3459e88b466ea711ed536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d"
I20260925 22:54:07.019706 27811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:07.021029 27811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f05abc8ef3459e88b466ea711ed536 in term 1.
I20260925 22:54:07.021468 27636 leader_election.cc:302] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68f05abc8ef3459e88b466ea711ed536, ce9f3acdfe004a0ebdfd071ff7e3d77d; no voters:
I20260925 22:54:07.021920 27660 leader_election.cc:288] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [CANDIDATE]: Term 1 election: Requested vote from peers 6336d051c0654888b4c5918b03a58ded (127.4.219.189:41973), ce9f3acdfe004a0ebdfd071ff7e3d77d (127.4.219.188:37879)
I20260925 22:54:07.021857 27847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:54:07.022449 27847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [term 1 LEADER]: Becoming Leader. State: Replica: 68f05abc8ef3459e88b466ea711ed536, State: Running, Role: LEADER
I20260925 22:54:07.022790 27847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.022964 27761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.023157 27761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6336d051c0654888b4c5918b03a58ded, State: Initialized, Role: FOLLOWER
I20260925 22:54:07.023464 27761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } }
I20260925 22:54:07.024991 27761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:54:07.024961 27871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f05abc8ef3459e88b466ea711ed536. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.025084 27871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:07.025069 27734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68f05abc8ef3459e88b466ea711ed536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6336d051c0654888b4c5918b03a58ded"
I20260925 22:54:07.025250 27734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:54:07.020761 27618 master_runner.cc:186] Error getting master registration for 127.4.219.189:41973: OK, instance_id { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" instance_seqno: 1790376846740054 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:54:07.026597 27618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:54:07.026695 27734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68f05abc8ef3459e88b466ea711ed536 in term 1.
W20260925 22:54:07.030601 27889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:07.030676 27889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 22:54:07.031978 27859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:07.032044 27859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:07.032104 27878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:54:07.032968 27878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 22:54:07.033581 27870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:54:07.033659 27870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:54:07.035002 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.035120 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.036794 27872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.036980 27872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.038431 27811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [term 1 FOLLOWER]: Refusing update from remote peer 68f05abc8ef3459e88b466ea711ed536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:07.039062 27871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:07.043780 27734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [term 1 FOLLOWER]: Refusing update from remote peer 68f05abc8ef3459e88b466ea711ed536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:54:07.044238 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f05abc8ef3459e88b466ea711ed536. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.044342 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.044317 27871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:54:07.047870 27843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:07.028424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.4.219.188:37879)
0925 22:54:07.028627 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.028632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.037597 (+ 8965us) server_negotiation.cc:407] Connection header received
0925 22:54:07.037755 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.037758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.037806 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.037856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.038868 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.039645 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.040900 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.041274 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.041351 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:07.043234 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:07.043238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:07.043240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:07.043265 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:07.045961 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.046467 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.046471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.046473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.046559 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.046889 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.046892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.046894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.047112 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:07.047214 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:07.047591 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 22:54:07.047712 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 22:54:07.048202 27872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f05abc8ef3459e88b466ea711ed536. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.048338 27872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.054821 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:07.025630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.4.219.190:44781)
0925 22:54:07.025928 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.025933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.037416 (+ 11483us) server_negotiation.cc:407] Connection header received
0925 22:54:07.037461 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.037466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.037515 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.037595 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.043644 (+ 6049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.044477 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.046968 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.047297 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.047372 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:07.049403 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:07.049407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:07.049409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:07.049438 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:07.052806 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.053281 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.053284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.053286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.053386 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.053856 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.053860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.053862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.054076 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:07.054200 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:07.054517 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 22:54:07.054653 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260925 22:54:07.055088 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:07.026929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.4.219.189:41973)
0925 22:54:07.027035 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.027040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.031051 (+ 4011us) server_negotiation.cc:407] Connection header received
0925 22:54:07.031246 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.031249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.031293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.031339 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.036673 (+ 5334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.037407 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.048624 (+ 11217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:07.049023 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:07.049111 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:07.051300 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:54:07.051316 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:54:07.051319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:54:07.051346 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:54:07.053082 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.053573 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.053577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.053578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.053662 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:54:07.053953 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:54:07.053957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:54:07.053959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:54:07.054162 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:54:07.054278 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:07.054823 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 22:54:07.054943 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:54:07.058923 27871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.059041 27871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:07.060062 27872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.060174 27872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.061655 27896 mvcc.cc:205] Tried to move back new op lower bound from 7333383565460602880 to 7333383565408698368. Current Snapshot: MvccSnapshot[applied={T|T < 7333383565460602880}]
I20260925 22:54:07.061991 27899 mvcc.cc:205] Tried to move back new op lower bound from 7333383565460602880 to 7333383565408698368. Current Snapshot: MvccSnapshot[applied={T|T < 7333383565460602880}]
I20260925 22:54:07.066102 27878 catalog_manager.cc:1347] Generated new cluster ID: 5cbb491fc5c74e5eb831f14fe543bf66
I20260925 22:54:07.066169 27878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:54:07.071983 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.072103 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536 [sys.catalog]: This master's current role is: LEADER
I20260925 22:54:07.073263 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68f05abc8ef3459e88b466ea711ed536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f05abc8ef3459e88b466ea711ed536" member_type: VOTER last_known_addr { host: "127.4.219.190" port: 44781 } } peers { permanent_uuid: "6336d051c0654888b4c5918b03a58ded" member_type: VOTER last_known_addr { host: "127.4.219.189" port: 41973 } } peers { permanent_uuid: "ce9f3acdfe004a0ebdfd071ff7e3d77d" member_type: VOTER last_known_addr { host: "127.4.219.188" port: 37879 } } }
I20260925 22:54:07.073395 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:54:07.073879 27897 mvcc.cc:205] Tried to move back new op lower bound from 7333383565460602880 to 7333383565408698368. Current Snapshot: MvccSnapshot[applied={T|T < 7333383565460602880}]
I20260925 22:54:07.082058 27878 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:54:07.083355 27878 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:54:07.098547 27878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f05abc8ef3459e88b466ea711ed536: Generated new TSK 0
I20260925 22:54:07.099447 27878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:54:07.128176 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.125110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.4.219.189:41973)
0925 22:54:07.125235 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.125240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.127050 (+ 1810us) server_negotiation.cc:407] Connection header received
0925 22:54:07.127244 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.127248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.127296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.127361 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.128040 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34794: BlockingRecv error: recv got EOF from 127.0.0.1:34794 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:54:07.129614 27689 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.125870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.4.219.190:44781)
0925 22:54:07.125977 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.125981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.128559 (+ 2578us) server_negotiation.cc:407] Connection header received
0925 22:54:07.128880 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.128883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.128936 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.128987 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.129504 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55884: BlockingRecv error: recv got EOF from 127.0.0.1:55884 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 22:54:07.130867 27843 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.126460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.4.219.188:37879)
0925 22:54:07.126575 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.126579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.129962 (+ 3383us) server_negotiation.cc:407] Connection header received
0925 22:54:07.130170 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.130173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.130219 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.130267 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.130751 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36240: BlockingRecv error: recv got EOF from 127.0.0.1:36240 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:54:07.135165 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.133716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.4.219.189:41973)
0925 22:54:07.133811 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.133815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.134265 (+ 450us) server_negotiation.cc:407] Connection header received
0925 22:54:07.134492 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.134495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.134570 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.134619 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.135072 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34804: BlockingRecv error: recv got EOF from 127.0.0.1:34804 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 22:54:07.139703 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.138364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.4.219.189:41973)
0925 22:54:07.138458 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.138461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.138917 (+ 456us) server_negotiation.cc:407] Connection header received
0925 22:54:07.139083 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.139086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.139130 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.139179 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.139616 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34808: BlockingRecv error: recv got EOF from 127.0.0.1:34808 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 22:54:07.145690 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.144317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.4.219.189:41973)
0925 22:54:07.144405 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.144409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.144929 (+ 520us) server_negotiation.cc:407] Connection header received
0925 22:54:07.145089 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.145092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.145135 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.145180 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.145609 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34820: BlockingRecv error: recv got EOF from 127.0.0.1:34820 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 22:54:07.153234 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.151430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34822 (local address 127.4.219.189:41973)
0925 22:54:07.151552 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.151556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.152038 (+ 482us) server_negotiation.cc:407] Connection header received
0925 22:54:07.152209 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.152212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.152256 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.152447 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.153137 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34822: BlockingRecv error: recv got EOF from 127.0.0.1:34822 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 22:54:07.165182 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.163417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.4.219.189:41973)
0925 22:54:07.163545 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.163550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.164131 (+ 581us) server_negotiation.cc:407] Connection header received
0925 22:54:07.164350 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.164354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.164407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.164460 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.165069 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34824: BlockingRecv error: recv got EOF from 127.0.0.1:34824 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:54:07.185628 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.183681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.4.219.189:41973)
0925 22:54:07.183832 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.183837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.184488 (+ 651us) server_negotiation.cc:407] Connection header received
0925 22:54:07.184761 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.184777 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.184838 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.184899 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.185502 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34836: BlockingRecv error: recv got EOF from 127.0.0.1:34836 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:54:07.222662 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.220570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.4.219.189:41973)
0925 22:54:07.220725 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.220730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.221503 (+ 773us) server_negotiation.cc:407] Connection header received
0925 22:54:07.221764 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.221768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.221830 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.221889 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.222534 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34848: BlockingRecv error: recv got EOF from 127.0.0.1:34848 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:54:07.290438 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.289310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.4.219.189:41973)
0925 22:54:07.289473 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.289477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.289595 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:54:07.289687 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.289690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.289764 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.289849 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.290284 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34862: BlockingRecv error: recv got EOF from 127.0.0.1:34862 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 22:54:07.422152 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.420735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.4.219.189:41973)
0925 22:54:07.420903 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.420908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.421114 (+ 206us) server_negotiation.cc:407] Connection header received
0925 22:54:07.421375 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.421379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.421442 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.421542 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.422027 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34876: BlockingRecv error: recv got EOF from 127.0.0.1:34876 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:54:07.681624 27767 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:54:07.680474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.4.219.189:41973)
0925 22:54:07.680632 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:07.680636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:07.680795 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:54:07.680918 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:07.680920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:07.680967 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:07.681046 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:54:07.681488 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34882: BlockingRecv error: recv got EOF from 127.0.0.1:34882 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:54:08.031946 27889 catalog_manager.cc:1259] Loaded cluster ID: 5cbb491fc5c74e5eb831f14fe543bf66
I20260925 22:54:08.032033 27889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: loading cluster ID for follower catalog manager: success
I20260925 22:54:08.033797 27889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: acquiring CA information for follower catalog manager: success
I20260925 22:54:08.034832 27870 catalog_manager.cc:1259] Loaded cluster ID: 5cbb491fc5c74e5eb831f14fe543bf66
I20260925 22:54:08.034927 27870 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: loading cluster ID for follower catalog manager: success
I20260925 22:54:08.035070 27889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6336d051c0654888b4c5918b03a58ded: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:54:08.036459 27870 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: acquiring CA information for follower catalog manager: success
I20260925 22:54:08.037498 27870 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce9f3acdfe004a0ebdfd071ff7e3d77d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:54:08.200208 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:08.196005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.4.219.189:41973)
0925 22:54:08.196251 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:08.196256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:08.196394 (+ 138us) server_negotiation.cc:407] Connection header received
0925 22:54:08.196546 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:08.196549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:08.196679 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:08.196812 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:08.197051 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.197618 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.198547 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.198812 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.198901 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:08.199251 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:08.199713 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:08.199844 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:08.199908 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:54:08.199970 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260925 22:54:08.207301 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:08.203478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.4.219.188:37879)
0925 22:54:08.203750 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:08.203753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:08.203775 (+ 22us) server_negotiation.cc:407] Connection header received
0925 22:54:08.203915 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:08.203917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:08.204032 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:08.204133 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:08.204333 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.204937 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.205723 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.205953 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.206031 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:08.206400 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:08.206862 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:08.206953 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:08.207051 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:54:08.207097 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260925 22:54:08.214869 27970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:08.210567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34898 (local address 127.4.219.189:41973)
0925 22:54:08.210694 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:08.210698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:08.211126 (+ 428us) server_negotiation.cc:407] Connection header received
0925 22:54:08.211284 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:08.211287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:08.211346 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:08.211436 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:08.211773 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.212376 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.213508 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.213856 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.213955 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:08.214291 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:08.214396 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:08.214463 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:08.214618 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 22:54:08.214678 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:54:08.215037 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:08.210917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.4.219.188:37879)
0925 22:54:08.211031 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:08.211035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:08.211421 (+ 386us) server_negotiation.cc:407] Connection header received
0925 22:54:08.211563 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:08.211567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:08.211613 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:08.211660 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:08.212036 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.212776 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.213920 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.214262 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.214363 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:08.214598 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:08.214671 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:08.214740 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:08.214860 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 22:54:08.214919 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:54:08.216288 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:54:08.210502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.4.219.190:44781)
0925 22:54:08.211013 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 22:54:08.211018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:54:08.211032 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:54:08.211135 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:54:08.211139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:54:08.211327 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:54:08.211545 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:54:08.212498 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.213218 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.214197 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:54:08.214547 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:54:08.214641 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:54:08.215466 (+ 825us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:54:08.215887 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:54:08.215979 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:54:08.216081 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 22:54:08.216145 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":66,"threads_started":1}
I20260925 22:54:08.219460 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27618
I20260925 22:54:08.226182 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27693
I20260925 22:54:08.232270 4974 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskre8WLI/build/debug/bin/kudu with pid 27768
2026-09-25T22:54:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3046 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6152 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281304 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST